00000100:00100000:3.0:1788201209.139149:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1263 00000100:00100000:3.0:1788201209.139155:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+5:13449:x1875064458197632:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.139163:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cff88b7/0x4605c7c97e1c32e2 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x3c7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c6630 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.139183:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/7 locks to priority blp list, 7 blwis in pool 00000100:00100000:3.0:1788201209.139187:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+5:13449:x1875064458197632:12345-192.168.201.130@tcp:104: Request processed in 32us (264us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.139191:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1263 00000100:00100000:3.0:1788201209.141784:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.141813:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458197888 00000100:00100000:3.0:1788201209.141821:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458197888 00000100:00100000:3.0:1788201209.141865:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1264 00000100:00100000:3.0:1788201209.141874:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13448:x1875064458197888:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.141912:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007560b9b8/0x4605c7c97e1beec5 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x2d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4ba150 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.141974:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201209.141981:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13448:x1875064458197888:12345-192.168.201.130@tcp:104: Request processed in 108us (205us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.141988:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1264 00000100:00100000:3.0:1788201209.146254:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.146304:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458198144 00000100:00100000:3.0:1788201209.146315:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458198144 00000100:00100000:3.0:1788201209.146369:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1265 00000100:00100000:3.0:1788201209.146379:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13447:x1875064458198144:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.146402:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000fd28ca48/0x4605c7c97e1c0c5b lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x33b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4bf6a0 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.146497:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201209.146504:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13447:x1875064458198144:12345-192.168.201.130@tcp:104: Request processed in 126us (267us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.148488:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1265 00000100:00100000:3.0:1788201209.158841:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.158918:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.158945:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.158991:0:11886:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1788201209.159020:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458198784 00000100:00100000:3.0:1788201209.159033:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458198784 00000100:00100000:3.0:1788201209.159099:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1266 00000100:00100000:3.0:1788201209.159109:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902342dfb050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6705:x1875064458198784:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.159134:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000055e9b48d/0x4605c7c97e1c2c44 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x3b0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c537c expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.159211:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201209.159217:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902342dfb050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6705:x1875064458198784:12345-192.168.201.130@tcp:104: Request processed in 110us (391us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.159225:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1266 00000100:00100000:3.0:1788201209.159272:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.159403:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.159433:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458199808 00000100:00100000:3.0:1788201209.159437:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458199808 00000100:00100000:3.0:1788201209.159453:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458199936 00000100:00100000:3.0:1788201209.159455:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458199936 00000100:00100000:3.0:1788201209.159466:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1267 00000100:00100000:3.0:1788201209.159473:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:12363:x1875064458199808:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.159484:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e5e5e863/0x4605c7c97e1bd3e4 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x26e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b538c expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201209.159575:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1788201209.159581:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:12363:x1875064458199808:12345-192.168.201.130@tcp:104: Request processed in 109us (308us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.159587:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1267 00000100:00100000:3.0:1788201209.159600:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1268 00000100:00100000:3.0:1788201209.159606:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13441:x1875064458199936:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.159615:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e535edbc/0x4605c7c97e1c3f6f lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x3f4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c8a25 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.159640:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to priority blp list, 3 blwis in pool 00000100:00100000:3.0:1788201209.159645:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13441:x1875064458199936:12345-192.168.201.130@tcp:104: Request processed in 39us (243us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.159650:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1268 00010000:00010000:3.0:1788201209.159926:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:3.0:1788201209.159935:0:5598:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000055e9b48d/0x4605c7c97e1c2c44 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x3b0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c537c expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.159945:0:5598:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9023450e3600 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201209.159951:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000055e9b48d/0x4605c7c97e1c2c44 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x3b0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4c537c expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.160051:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000055e9b48d/0x4605c7c97e1c2c44 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x3b0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c537c expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.160060:0:5598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201209.160084:0:5598:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000055e9b48d/0x4605c7c97e1c2c44 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x3b0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c537c expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.160093:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000055e9b48d/0x4605c7c97e1c2c44 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x3b0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c537c expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.160116:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1788201209.160121:0:5598:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e5e5e863/0x4605c7c97e1bd3e4 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x26e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b538c expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201209.160127:0:5598:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235246ce00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201209.160130:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e5e5e863/0x4605c7c97e1bd3e4 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x26e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4b538c expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201209.160164:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e5e5e863/0x4605c7c97e1bd3e4 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x26e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b538c expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201209.160171:0:5598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201209.160178:0:5598:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e5e5e863/0x4605c7c97e1bd3e4 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x26e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b538c expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201209.160186:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e5e5e863/0x4605c7c97e1bd3e4 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x26e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b538c expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201209.160208:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201209.160211:0:5598:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e535edbc/0x4605c7c97e1c3f6f lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x3f4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c8a25 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.160218:0:5598:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345156e00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201209.160222:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e535edbc/0x4605c7c97e1c3f6f lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x3f4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4c8a25 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.160253:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e535edbc/0x4605c7c97e1c3f6f lrc: 2/0,0 mode: --/PR res: [0x280000400:0x3f4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c8a25 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.160261:0:5598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201209.160265:0:5598:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e535edbc/0x4605c7c97e1c3f6f lrc: 1/0,0 mode: --/PR res: [0x280000400:0x3f4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c8a25 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.160273:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e535edbc/0x4605c7c97e1c3f6f lrc: 0/0,0 mode: --/PR res: [0x280000400:0x3f4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c8a25 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.160287:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.160289:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.160291:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.164984:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201209.186127:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1274 00000100:00100000:2.0:1788201209.186137:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c4000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+6:13439:x1875064458201600:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201209.186160:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e5a07383/0x4605c7c97e1c3e7a lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x3f1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c8746 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186243:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201209.186250:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c4000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+6:13439:x1875064458201600:12345-192.168.201.130@tcp:104: Request processed in 114us (446us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201209.186258:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1274 00000100:00100000:2.0:1788201209.186275:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1275 00000100:00100000:2.0:1788201209.186281:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c5880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13448:x1875064458201728:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201209.186291:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000055be9d01/0x4605c7c97e1bedfa lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x2cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b9eef expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186320:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:2.0:1788201209.186326:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c5880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13448:x1875064458201728:12345-192.168.201.130@tcp:104: Request processed in 44us (459us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201209.186332:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1275 00000100:00100000:2.0:1788201209.186343:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1276 00000100:00100000:2.0:1788201209.186349:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c4a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13449:x1875064458201856:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201209.186358:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000000a72c5f6/0x4605c7c97e1bbee4 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b16d8 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186384:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to priority blp list, 3 blwis in pool 00000100:00100000:2.0:1788201209.186390:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c4a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13449:x1875064458201856:12345-192.168.201.130@tcp:104: Request processed in 41us (474us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201209.186395:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1276 00000100:00100000:2.0:1788201209.186419:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1277 00000100:00100000:2.0:1788201209.186425:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c5500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13446:x1875064458201984:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201209.186434:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000049dada27/0x4605c7c97e1bdf8a lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x299:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b7558 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186462:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/4 locks to priority blp list, 4 blwis in pool 00000100:00100000:2.0:1788201209.186467:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c5500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13446:x1875064458201984:12345-192.168.201.130@tcp:104: Request processed in 42us (531us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201209.186473:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1277 00000100:00100000:2.0:1788201209.186484:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1278 00000100:00100000:2.0:1788201209.186489:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c7480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6703:x1875064458202240:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201209.186498:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000062fdfa26/0x4605c7c97e1c1798 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x364:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c1731 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186520:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/5 locks to priority blp list, 5 blwis in pool 00000100:00100000:2.0:1788201209.186524:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c7480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6703:x1875064458202240:12345-192.168.201.130@tcp:104: Request processed in 35us (543us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201209.186529:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1278 00010000:00010000:2.0:1788201209.186547:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 4 total in blp. (4 blwis in pool) 00010000:00010000:2.0:1788201209.186554:0:5598:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e5a07383/0x4605c7c97e1c3e7a lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x3f1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c8746 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186564:0:5598:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9023533e8a00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201209.186570:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e5a07383/0x4605c7c97e1c3e7a lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x3f1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4c8746 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186677:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e5a07383/0x4605c7c97e1c3e7a lrc: 2/0,0 mode: --/PR res: [0x240000400:0x3f1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c8746 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186686:0:5598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201209.186711:0:5598:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e5a07383/0x4605c7c97e1c3e7a lrc: 1/0,0 mode: --/PR res: [0x240000400:0x3f1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c8746 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186720:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e5a07383/0x4605c7c97e1c3e7a lrc: 0/0,0 mode: --/PR res: [0x240000400:0x3f1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c8746 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186741:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 3 total in blp. (3 blwis in pool) 00010000:00010000:2.0:1788201209.186745:0:5598:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000055be9d01/0x4605c7c97e1bedfa lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x2cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b9eef expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186752:0:5598:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351e41600 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201209.186755:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000055be9d01/0x4605c7c97e1bedfa lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x2cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4b9eef expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186787:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000055be9d01/0x4605c7c97e1bedfa lrc: 2/0,0 mode: --/PR res: [0x240000400:0x2cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b9eef expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186794:0:5598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201209.186800:0:5598:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000055be9d01/0x4605c7c97e1bedfa lrc: 1/0,0 mode: --/PR res: [0x240000400:0x2cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b9eef expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186814:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000055be9d01/0x4605c7c97e1bedfa lrc: 0/0,0 mode: --/PR res: [0x240000400:0x2cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b9eef expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186825:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:2.0:1788201209.186828:0:5598:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000000a72c5f6/0x4605c7c97e1bbee4 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b16d8 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186835:0:5598:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9023436eb400 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201209.186838:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000000a72c5f6/0x4605c7c97e1bbee4 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4b16d8 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186866:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000000a72c5f6/0x4605c7c97e1bbee4 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b16d8 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186872:0:5598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201209.186903:0:5598:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000000a72c5f6/0x4605c7c97e1bbee4 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b16d8 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186911:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000000a72c5f6/0x4605c7c97e1bbee4 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x222:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b16d8 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186925:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1788201209.186928:0:5598:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000049dada27/0x4605c7c97e1bdf8a lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x299:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b7558 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186940:0:5598:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902350c6d400 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201209.186944:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000049dada27/0x4605c7c97e1bdf8a lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x299:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4b7558 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186972:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000049dada27/0x4605c7c97e1bdf8a lrc: 2/0,0 mode: --/PR res: [0x240000400:0x299:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b7558 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186979:0:5598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201209.186983:0:5598:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000049dada27/0x4605c7c97e1bdf8a lrc: 1/0,0 mode: --/PR res: [0x240000400:0x299:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b7558 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.186991:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000049dada27/0x4605c7c97e1bdf8a lrc: 0/0,0 mode: --/PR res: [0x240000400:0x299:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b7558 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.187003:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201209.187006:0:5598:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000062fdfa26/0x4605c7c97e1c1798 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x364:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c1731 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.187012:0:5598:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234368a600 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201209.187016:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000062fdfa26/0x4605c7c97e1c1798 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x364:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4c1731 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.187041:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000062fdfa26/0x4605c7c97e1c1798 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x364:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c1731 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.187053:0:5598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201209.187057:0:5598:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000062fdfa26/0x4605c7c97e1c1798 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x364:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c1731 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.187064:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000062fdfa26/0x4605c7c97e1c1798 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x364:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c1731 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.187075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201209.187077:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201209.187079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201209.187086:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201209.187107:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442604288:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.187168:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442604544:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.187189:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:2.0:1788201209.187194:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442604160:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.187209:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442604416:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.187224:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442604672:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201209.187257:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201209.191516:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201209.193991:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.194033:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458202496 00000100:00100000:3.0:1788201209.194046:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458202496 00000100:00100000:3.0:1788201209.194107:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1279 00000100:00100000:3.0:1788201209.194120:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13441:x1875064458202496:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.194147:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000440ea656/0x4605c7c97e1c1052 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x34a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c0285 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.194236:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201209.194243:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13441:x1875064458202496:12345-192.168.201.130@tcp:104: Request processed in 126us (267us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.194251:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1279 00010000:00010000:3.0:1788201209.194278:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201209.194285:0:5598:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000440ea656/0x4605c7c97e1c1052 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x34a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c0285 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.194295:0:5598:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902353081400 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201209.194302:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000440ea656/0x4605c7c97e1c1052 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x34a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4c0285 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.194402:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000440ea656/0x4605c7c97e1c1052 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x34a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c0285 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.194423:0:5598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201209.194442:0:5598:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000440ea656/0x4605c7c97e1c1052 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x34a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c0285 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.194451:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000440ea656/0x4605c7c97e1c1052 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x34a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c0285 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.194483:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.194485:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.194486:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201209.195121:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201209.195145:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442604800:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.195359:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442602880:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.195377:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442602752:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.195386:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442603008:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201209.196415:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.196548:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.196581:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.202426:0:5590:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1788201209.202458:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458202880 00000100:00100000:3.0:1788201209.202469:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458202880 00000100:00100000:3.0:1788201209.202545:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1280 00000100:00100000:3.0:1788201209.202558:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023451bb450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6705:x1875064458202880:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.202756:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000000747093/0x4605c7c97e1be4b5 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x2ab:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b8446 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.202849:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201209.202857:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023451bb450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6705:x1875064458202880:12345-192.168.201.130@tcp:104: Request processed in 301us (6461us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.202865:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1280 00010000:00010000:3.0:1788201209.202934:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201209.202942:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000000747093/0x4605c7c97e1be4b5 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x2ab:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b8446 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.202952:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342e16600 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201209.202959:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000000747093/0x4605c7c97e1be4b5 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x2ab:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4b8446 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.203061:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000000747093/0x4605c7c97e1be4b5 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x2ab:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b8446 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.203068:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201209.203088:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000000747093/0x4605c7c97e1be4b5 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x2ab:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b8446 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.203095:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000000747093/0x4605c7c97e1be4b5 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x2ab:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b8446 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.203117:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.203119:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.203120:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201209.203419:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442604928:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.203516:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442603392:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.203529:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442603648:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201209.204263:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201209.215808:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201209.215855:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201209.215929:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458203520 00000100:00100000:2.0:1788201209.215943:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458203520 00000100:00100000:2.0:1788201209.216002:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458203776 00000100:00100000:2.0:1788201209.216006:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458203776 00000100:00100000:2.0:1788201209.216025:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1281 00000100:00100000:2.0:1788201209.216037:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c4e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13440:x1875064458203520:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201209.216062:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003c1bf68d/0x4605c7c97e1be6ec lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x2b3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b8a58 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.216162:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201209.216169:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c4e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13440:x1875064458203520:12345-192.168.201.130@tcp:104: Request processed in 134us (384us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201209.216177:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1281 00000100:00100000:2.0:1788201209.216194:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1282 00000100:00100000:2.0:1788201209.216201:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c5f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13445:x1875064458203776:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201209.216213:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d5b07e60/0x4605c7c97e1bf6c8 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x2ed:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4bb833 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.216239:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:2.0:1788201209.216243:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c5f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13445:x1875064458203776:12345-192.168.201.130@tcp:104: Request processed in 44us (388us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201209.216249:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1282 00010000:00010000:3.0:1788201209.217758:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1788201209.218364:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003c1bf68d/0x4605c7c97e1be6ec lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x2b3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b8a58 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.218374:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235864c200 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201209.218379:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003c1bf68d/0x4605c7c97e1be6ec lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x2b3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4b8a58 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.218453:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003c1bf68d/0x4605c7c97e1be6ec lrc: 2/0,0 mode: --/PR res: [0x280000400:0x2b3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b8a58 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.218461:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201209.218477:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003c1bf68d/0x4605c7c97e1be6ec lrc: 1/0,0 mode: --/PR res: [0x280000400:0x2b3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b8a58 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.218485:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003c1bf68d/0x4605c7c97e1be6ec lrc: 0/0,0 mode: --/PR res: [0x280000400:0x2b3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b8a58 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.218503:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.218505:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.218506:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.218515:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201209.218781:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.218806:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.218824:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.218850:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.218866:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.218910:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.218938:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.218952:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.219018:0:5591:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1788201209.219032:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458204160 00000100:00100000:3.0:1788201209.219040:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458204160 00000100:00100000:3.0:1788201209.219072:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458204288 00000100:00100000:3.0:1788201209.219074:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458204288 00000100:00100000:3.0:1788201209.219082:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458204544 00000100:00100000:3.0:1788201209.219084:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458204544 00000100:00100000:3.0:1788201209.219091:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458205056 00000100:00100000:3.0:1788201209.219093:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458205056 00000100:00100000:3.0:1788201209.219102:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458205184 00000100:00100000:3.0:1788201209.219104:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458205184 00000100:00100000:3.0:1788201209.219116:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1283 00000100:00100000:3.0:1788201209.219126:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+6:13448:x1875064458204160:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.219140:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006b878494/0x4605c7c97e1bc47f lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x237:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b2716 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219189:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201209.219195:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+6:13448:x1875064458204160:12345-192.168.201.130@tcp:104: Request processed in 70us (417us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.219202:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1283 00000100:00100000:3.0:1788201209.219218:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1284 00000100:00100000:3.0:1788201209.219223:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13443:x1875064458204288:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.219231:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000003df829c/0x4605c7c97e1bd2ef lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x26b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b50ad expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201209.219256:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1788201209.219260:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13443:x1875064458204288:12345-192.168.201.130@tcp:104: Request processed in 38us (455us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.219266:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1284 00000100:00100000:3.0:1788201209.219276:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1285 00000100:00100000:3.0:1788201209.219281:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6706:x1875064458204544:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.219289:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001c542f6b/0x4605c7c97e1bc574 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x23a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b2962 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219310:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to priority blp list, 3 blwis in pool 00000100:00100000:3.0:1788201209.219315:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6706:x1875064458204544:12345-192.168.201.130@tcp:104: Request processed in 34us (492us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.219321:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1285 00000100:00100000:3.0:1788201209.219330:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1286 00000100:00100000:3.0:1788201209.219335:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13450:x1875064458205056:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.219343:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a9bc7eb3/0x4605c7c97e1c434a lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x402:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c9523 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219364:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/4 locks to priority blp list, 4 blwis in pool 00000100:00100000:3.0:1788201209.219369:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13450:x1875064458205056:12345-192.168.201.130@tcp:104: Request processed in 34us (518us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.219374:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1286 00000100:00100000:3.0:1788201209.219383:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1287 00000100:00100000:3.0:1788201209.219388:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902342dfb450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6703:x1875064458205184:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.219396:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a4a1dbf6/0x4605c7c97e1c23f4 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c3b1f expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219430:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/5 locks to priority blp list, 5 blwis in pool 00000100:00100000:3.0:1788201209.219435:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902342dfb450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6703:x1875064458205184:12345-192.168.201.130@tcp:104: Request processed in 47us (569us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.219441:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1287 00010000:00010000:3.0:1788201209.219467:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 4 total in blp. (4 blwis in pool) 00010000:00010000:3.0:1788201209.219472:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006b878494/0x4605c7c97e1bc47f lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x237:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b2716 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219480:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352647a00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201209.219484:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006b878494/0x4605c7c97e1bc47f lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x237:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4b2716 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219537:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006b878494/0x4605c7c97e1bc47f lrc: 2/0,0 mode: --/PR res: [0x240000400:0x237:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b2716 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219544:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201209.219552:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006b878494/0x4605c7c97e1bc47f lrc: 1/0,0 mode: --/PR res: [0x240000400:0x237:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b2716 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219559:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006b878494/0x4605c7c97e1bc47f lrc: 0/0,0 mode: --/PR res: [0x240000400:0x237:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b2716 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219576:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 3 total in blp. (3 blwis in pool) 00010000:00010000:3.0:1788201209.219579:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000003df829c/0x4605c7c97e1bd2ef lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x26b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b50ad expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201209.219586:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235246c400 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201209.219590:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000003df829c/0x4605c7c97e1bd2ef lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x26b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4b50ad expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201209.219618:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000003df829c/0x4605c7c97e1bd2ef lrc: 2/0,0 mode: --/PR res: [0x240000400:0x26b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b50ad expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201209.219626:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201209.219630:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000003df829c/0x4605c7c97e1bd2ef lrc: 1/0,0 mode: --/PR res: [0x240000400:0x26b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b50ad expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201209.219644:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000003df829c/0x4605c7c97e1bd2ef lrc: 0/0,0 mode: --/PR res: [0x240000400:0x26b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b50ad expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201209.219658:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:3.0:1788201209.219661:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001c542f6b/0x4605c7c97e1bc574 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x23a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b2962 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219667:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9023821f6c00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201209.219670:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001c542f6b/0x4605c7c97e1bc574 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x23a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4b2962 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219694:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001c542f6b/0x4605c7c97e1bc574 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x23a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b2962 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219701:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201209.219705:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001c542f6b/0x4605c7c97e1bc574 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x23a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b2962 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219711:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001c542f6b/0x4605c7c97e1bc574 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x23a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b2962 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219725:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1788201209.219728:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a9bc7eb3/0x4605c7c97e1c434a lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x402:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c9523 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219739:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234583c600 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201209.219743:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a9bc7eb3/0x4605c7c97e1c434a lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x402:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4c9523 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219769:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a9bc7eb3/0x4605c7c97e1c434a lrc: 2/0,0 mode: --/PR res: [0x280000400:0x402:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c9523 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219776:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201209.219780:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a9bc7eb3/0x4605c7c97e1c434a lrc: 1/0,0 mode: --/PR res: [0x280000400:0x402:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c9523 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219788:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a9bc7eb3/0x4605c7c97e1c434a lrc: 0/0,0 mode: --/PR res: [0x280000400:0x402:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c9523 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219814:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201209.219817:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a4a1dbf6/0x4605c7c97e1c23f4 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c3b1f expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219824:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235005ee00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201209.219827:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a4a1dbf6/0x4605c7c97e1c23f4 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4c3b1f expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219858:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a4a1dbf6/0x4605c7c97e1c23f4 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c3b1f expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219870:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201209.219874:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a4a1dbf6/0x4605c7c97e1c23f4 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c3b1f expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219903:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a4a1dbf6/0x4605c7c97e1c23f4 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x391:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c3b1f expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.219916:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.219918:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.219919:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.219927:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.219932:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.219943:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201209.220163:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442605440:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.220233:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442605696:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.220254:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442605952:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.220294:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442604160:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.220306:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442604416:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.220314:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442604672:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.220323:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442604288:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.220332:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442604544:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.220340:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442604032:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.220368:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:2.0:1788201209.220374:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442605568:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.220394:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442605824:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.220427:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442606080:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.220462:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442604928:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.220471:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442604800:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.220480:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442603520:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.220488:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442603776:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.220496:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442603904:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201209.221672:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:2.0:1788201209.221680:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004710f64e/0x4605c7c97e1bcd8c lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x257:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b4117 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201209.221689:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902343664000 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201209.221696:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004710f64e/0x4605c7c97e1bcd8c lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x257:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4b4117 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201209.221791:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004710f64e/0x4605c7c97e1bcd8c lrc: 2/0,0 mode: --/PR res: [0x280000400:0x257:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b4117 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201209.221799:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201209.221820:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004710f64e/0x4605c7c97e1bcd8c lrc: 1/0,0 mode: --/PR res: [0x280000400:0x257:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b4117 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201209.221830:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004710f64e/0x4605c7c97e1bcd8c lrc: 0/0,0 mode: --/PR res: [0x280000400:0x257:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b4117 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201209.221852:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1788201209.221856:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f47a9f0/0x4605c7c97e1c0e22 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x342:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4bfbf5 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.221863:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345133e00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201209.221868:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f47a9f0/0x4605c7c97e1c0e22 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x342:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4bfbf5 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.224077:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f47a9f0/0x4605c7c97e1c0e22 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x342:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bfbf5 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.224091:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201209.224107:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f47a9f0/0x4605c7c97e1c0e22 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x342:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bfbf5 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.224116:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f47a9f0/0x4605c7c97e1c0e22 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x342:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bfbf5 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.224140:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201209.224146:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c7951523/0x4605c7c97e1c1101 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x34d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c0492 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.224154:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902353099600 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201209.224160:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c7951523/0x4605c7c97e1c1101 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x34d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4c0492 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.224192:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c7951523/0x4605c7c97e1c1101 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x34d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c0492 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.224200:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201209.224205:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c7951523/0x4605c7c97e1c1101 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x34d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c0492 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.224213:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c7951523/0x4605c7c97e1c1101 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x34d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c0492 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.224227:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201209.224229:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201209.224231:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201209.224256:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442606336:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.224346:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788201209.224356:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442606208:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.224379:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442606464:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201209.224570:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.227582:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201209.227680:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.227784:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.227827:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.227863:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.228002:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201209.228098:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458206336 00000100:00100000:3.0:1788201209.228109:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458206336 00000100:00100000:3.0:1788201209.228156:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458206464 00000100:00100000:3.0:1788201209.228158:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458206464 00000100:00100000:3.0:1788201209.228168:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458206592 00000100:00100000:3.0:1788201209.228170:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458206592 00000100:00100000:3.0:1788201209.228177:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458206848 00000100:00100000:3.0:1788201209.228179:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458206848 00000100:00100000:3.0:1788201209.228187:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458207104 00000100:00100000:3.0:1788201209.228189:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458207104 00000100:00100000:3.0:1788201209.228204:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1288 00000100:00100000:3.0:1788201209.228213:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518c700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+7:13441:x1875064458206336:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.228237:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009f0c203c/0x4605c7c97e1c08c6 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x32e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4bec74 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.228307:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201209.228314:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518c700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+7:13441:x1875064458206336:12345-192.168.201.130@tcp:104: Request processed in 102us (642us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.228321:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1288 00000100:00100000:3.0:1788201209.228340:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1289 00000100:00100000:3.0:1788201209.228346:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13447:x1875064458206464:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.228357:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006f13ccf/0x4605c7c97e1c34a9 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x3ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c6b85 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.228388:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1788201209.228393:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13447:x1875064458206464:12345-192.168.201.130@tcp:104: Request processed in 48us (610us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.228399:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1289 00000100:00100000:3.0:1788201209.228606:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1290 00000100:00100000:3.0:1788201209.228614:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+6:13442:x1875064458206592:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.228624:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000036dcdeb/0x4605c7c97e1c2f3f lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x3ba:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c5bda expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.228657:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to priority blp list, 3 blwis in pool 00000100:00100000:3.0:1788201209.228663:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+6:13442:x1875064458206592:12345-192.168.201.130@tcp:104: Request processed in 51us (836us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.228669:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1290 00000100:00100000:3.0:1788201209.228681:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1291 00000100:00100000:3.0:1788201209.228688:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90235268aa00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:6705:x1875064458206848:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.228696:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000addd5052/0x4605c7c97e1bfb98 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x2fe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4bc610 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.228725:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/4 locks to priority blp list, 4 blwis in pool 00000100:00100000:3.0:1788201209.228730:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90235268aa00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:6705:x1875064458206848:12345-192.168.201.130@tcp:104: Request processed in 43us (867us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.228735:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1291 00000100:00100000:3.0:1788201209.228746:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1292 00000100:00100000:3.0:1788201209.228753:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90235268b100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13439:x1875064458207104:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201209.228762:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d6ade3a7/0x4605c7c97e1c488a lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x415:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4ca450 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.228785:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/5 locks to priority blp list, 5 blwis in pool 00000100:00100000:3.0:1788201209.228790:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90235268b100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13439:x1875064458207104:12345-192.168.201.130@tcp:104: Request processed in 39us (789us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201209.228796:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1292 00010000:00010000:3.0:1788201209.228812:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 4 total in blp. (4 blwis in pool) 00010000:00010000:3.0:1788201209.228819:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009f0c203c/0x4605c7c97e1c08c6 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x32e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4bec74 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.228829:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902343720e00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201209.228836:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009f0c203c/0x4605c7c97e1c08c6 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x32e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4bec74 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229184:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009f0c203c/0x4605c7c97e1c08c6 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x32e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bec74 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229192:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201209.229218:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009f0c203c/0x4605c7c97e1c08c6 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x32e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bec74 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229227:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009f0c203c/0x4605c7c97e1c08c6 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x32e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bec74 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229252:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 3 total in blp. (3 blwis in pool) 00010000:00010000:3.0:1788201209.229256:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006f13ccf/0x4605c7c97e1c34a9 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x3ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c6b85 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229263:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342ee7200 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201209.229266:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006f13ccf/0x4605c7c97e1c34a9 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x3ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4c6b85 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229300:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006f13ccf/0x4605c7c97e1c34a9 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x3ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c6b85 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229307:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201209.229313:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006f13ccf/0x4605c7c97e1c34a9 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x3ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c6b85 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229326:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006f13ccf/0x4605c7c97e1c34a9 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x3ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c6b85 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229341:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:3.0:1788201209.229344:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000036dcdeb/0x4605c7c97e1c2f3f lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x3ba:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c5bda expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229351:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352ede000 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201209.229354:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000036dcdeb/0x4605c7c97e1c2f3f lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x3ba:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4c5bda expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229383:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000036dcdeb/0x4605c7c97e1c2f3f lrc: 2/0,0 mode: --/PR res: [0x280000400:0x3ba:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c5bda expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229391:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201209.229396:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000036dcdeb/0x4605c7c97e1c2f3f lrc: 1/0,0 mode: --/PR res: [0x280000400:0x3ba:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c5bda expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229414:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000036dcdeb/0x4605c7c97e1c2f3f lrc: 0/0,0 mode: --/PR res: [0x280000400:0x3ba:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c5bda expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229430:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1788201209.229433:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000addd5052/0x4605c7c97e1bfb98 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x2fe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4bc610 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229446:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865b800 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201209.229450:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000addd5052/0x4605c7c97e1bfb98 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x2fe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4bc610 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229482:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000addd5052/0x4605c7c97e1bfb98 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x2fe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bc610 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229490:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201209.229494:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000addd5052/0x4605c7c97e1bfb98 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x2fe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bc610 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229502:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000addd5052/0x4605c7c97e1bfb98 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x2fe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bc610 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229514:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201209.229517:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d6ade3a7/0x4605c7c97e1c488a lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x415:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4ca450 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229523:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342d62600 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201209.229527:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d6ade3a7/0x4605c7c97e1c488a lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x415:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4ca450 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229556:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d6ade3a7/0x4605c7c97e1c488a lrc: 2/0,0 mode: --/PR res: [0x280000400:0x415:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4ca450 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229569:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201209.229574:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d6ade3a7/0x4605c7c97e1c488a lrc: 1/0,0 mode: --/PR res: [0x280000400:0x415:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4ca450 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229582:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d6ade3a7/0x4605c7c97e1c488a lrc: 0/0,0 mode: --/PR res: [0x280000400:0x415:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4ca450 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201209.229594:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.229596:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.229598:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.229608:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.229621:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201209.233699:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201209.235790:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442606592:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.236011:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442606848:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.236036:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442607104:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.236068:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788201209.236075:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442606720:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.236097:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442606976:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.246247:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442606208:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.246269:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442606464:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.246277:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442606336:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.246313:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442605568:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.246322:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442605824:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.246329:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442606080:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.246337:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442605440:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.246345:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442605696:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.246351:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442605952:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.254662:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442606720:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.254690:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442606976:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.254703:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442606592:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.254713:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442606848:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.254723:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442607104:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.260161:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201209.260213:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201209.260313:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201209.260374:0:11886:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1788201209.260397:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458207488 00000100:00100000:2.0:1788201209.260424:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458207488 00000100:00100000:2.0:1788201209.260480:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458207616 00000100:00100000:2.0:1788201209.260484:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458207616 00000100:00100000:2.0:1788201209.260501:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1293 00000100:00100000:2.0:1788201209.260511:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c5c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:12363:x1875064458207488:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201209.260535:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a206f972/0x4605c7c97e1c4415 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x405:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c9799 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.261341:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:2.0:1788201209.261349:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c5c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:12363:x1875064458207488:12345-192.168.201.130@tcp:104: Request processed in 839us (1205us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201209.261356:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1293 00000100:00100000:2.0:1788201209.261371:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1294 00000100:00100000:2.0:1788201209.261377:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902342dfa450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13451:x1875064458207616:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201209.261386:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000009c41759/0x4605c7c97e1c07f4 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x32b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4be9fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.261438:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to priority blp list, 3 blwis in pool 00000100:00100000:2.0:1788201209.261444:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902342dfa450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13451:x1875064458207616:12345-192.168.201.130@tcp:104: Request processed in 67us (1231us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201209.261450:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1294 00010000:00010000:2.0:1788201209.261491:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:2.0:1788201209.261499:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a8dc9048/0x4605c7c97e1c2e6d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x3b7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c5964 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.261509:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352edf800 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201209.261516:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a8dc9048/0x4605c7c97e1c2e6d lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x3b7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4c5964 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.261613:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a8dc9048/0x4605c7c97e1c2e6d lrc: 2/0,0 mode: --/PR res: [0x280000400:0x3b7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c5964 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.261622:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201209.261645:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a8dc9048/0x4605c7c97e1c2e6d lrc: 1/0,0 mode: --/PR res: [0x280000400:0x3b7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c5964 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.261653:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a8dc9048/0x4605c7c97e1c2e6d lrc: 0/0,0 mode: --/PR res: [0x280000400:0x3b7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c5964 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.261676:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1788201209.261680:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a206f972/0x4605c7c97e1c4415 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x405:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c9799 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.261686:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351f59a00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201209.261689:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a206f972/0x4605c7c97e1c4415 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x405:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4c9799 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.261718:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a206f972/0x4605c7c97e1c4415 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x405:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c9799 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.261730:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201209.261734:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a206f972/0x4605c7c97e1c4415 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x405:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c9799 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.261742:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a206f972/0x4605c7c97e1c4415 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x405:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c9799 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.261754:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201209.261758:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000009c41759/0x4605c7c97e1c07f4 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x32b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4be9fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.261765:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902343721c00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201209.261769:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000009c41759/0x4605c7c97e1c07f4 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x32b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4be9fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.261800:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000009c41759/0x4605c7c97e1c07f4 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x32b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4be9fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.261807:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201209.261811:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000009c41759/0x4605c7c97e1c07f4 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x32b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4be9fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.261818:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000009c41759/0x4605c7c97e1c07f4 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x32b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4be9fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201209.261837:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201209.261839:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201209.261841:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201209.261861:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442607360:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.261958:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788201209.261965:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442607232:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.261989:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442607488:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201209.262428:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201209.275836:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442607232:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.275861:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442607488:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201209.275873:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442607360:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201210.245707:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201211.272763:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201213.313098:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201213.675486:0:13644:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 102a: check resend (request lost) with multiple modify RPCs in flight ========================================================== 14:33:32 (1788201212) 00000100:00100000:3.0:1788201213.723960:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201213.723998:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201213.724462:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458210048 00000100:00100000:2.0:1788201213.724473:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458210048 00000100:00100000:2.0:1788201213.724530:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458210176 00000100:00100000:2.0:1788201213.724533:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458210176 00000100:00100000:2.0:1788201213.724552:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1295 00000100:00100000:2.0:1788201213.724562:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902352688700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:13446:x1875064458210048:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201213.724581:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004b20a987/0x4605c7c97e1bfc6a lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x301:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4bc886 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201213.724663:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201213.724670:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902352688700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:13446:x1875064458210048:12345-192.168.201.130@tcp:104: Request processed in 109us (722us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201213.724678:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1295 00000100:00100000:2.0:1788201213.724693:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1296 00000100:00100000:2.0:1788201213.724700:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902352689180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13445:x1875064458210176:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201213.724904:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000005df50887/0x4605c7c97e1c1b6c lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x372:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c2298 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201213.724941:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:2.0:1788201213.724948:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902352689180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13445:x1875064458210176:12345-192.168.201.130@tcp:104: Request processed in 248us (948us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201213.724954:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1296 00010000:00010000:2.0:1788201213.725307:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1788201213.725313:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004b20a987/0x4605c7c97e1bfc6a lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x301:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4bc886 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201213.725323:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235277d000 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201213.725339:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004b20a987/0x4605c7c97e1bfc6a lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x301:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4bc886 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201213.725424:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004b20a987/0x4605c7c97e1bfc6a lrc: 2/0,0 mode: --/PR res: [0x280000400:0x301:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bc886 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201213.725442:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201213.725463:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004b20a987/0x4605c7c97e1bfc6a lrc: 1/0,0 mode: --/PR res: [0x280000400:0x301:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bc886 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201213.725471:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004b20a987/0x4605c7c97e1bfc6a lrc: 0/0,0 mode: --/PR res: [0x280000400:0x301:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bc886 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201213.725490:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201213.725494:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000005df50887/0x4605c7c97e1c1b6c lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x372:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c2298 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201213.725500:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352f3de00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201213.725504:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000005df50887/0x4605c7c97e1c1b6c lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x372:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4c2298 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201213.725534:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000005df50887/0x4605c7c97e1c1b6c lrc: 2/0,0 mode: --/PR res: [0x280000400:0x372:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c2298 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201213.725541:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201213.725547:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000005df50887/0x4605c7c97e1c1b6c lrc: 1/0,0 mode: --/PR res: [0x280000400:0x372:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c2298 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201213.725553:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000005df50887/0x4605c7c97e1c1b6c lrc: 0/0,0 mode: --/PR res: [0x280000400:0x372:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c2298 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201213.725563:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201213.725565:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201213.725566:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201213.725584:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442608128:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201213.725644:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201213.725650:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442608256:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201213.725809:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201213.729098:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201213.729134:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201213.729152:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201213.729388:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201213.729407:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201213.729447:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201213.729491:0:11886:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1788201213.729509:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458210304 00000100:00100000:3.0:1788201213.729519:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458210304 00000100:00100000:3.0:1788201213.729555:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458210432 00000100:00100000:3.0:1788201213.729601:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458210432 00000100:00100000:3.0:1788201213.729617:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458210560 00000100:00100000:3.0:1788201213.729621:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458210560 00000100:00100000:3.0:1788201213.729631:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458210688 00000100:00100000:3.0:1788201213.729633:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458210688 00000100:00100000:3.0:1788201213.729641:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458210816 00000100:00100000:3.0:1788201213.729643:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458210816 00000100:00100000:3.0:1788201213.729657:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1297 00000100:00100000:3.0:1788201213.729667:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90235268bb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+8:13450:x1875064458210304:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201213.729686:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000243b9a13/0x4605c7c97e1be055 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x29b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b7726 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.730080:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1788201213.730088:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90235268bb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+9:13450:x1875064458210304:12345-192.168.201.130@tcp:104: Request processed in 422us (1041us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201213.730096:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1297 00000100:00100000:3.0:1788201213.730115:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1298 00000100:00100000:3.0:1788201213.730122:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902352688380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+8:13448:x1875064458210432:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201213.730131:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf7dfa9c/0x4605c7c97e1c1da3 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x37a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c28bf expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.731590:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/0 locks to priority blp list, 0 blwis in pool 00000100:00100000:3.0:1788201213.731600:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902352688380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+7:13448:x1875064458210432:12345-192.168.201.130@tcp:104: Request processed in 1477us (2464us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201213.731609:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1298 00000100:00100000:3.0:1788201213.731633:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1299 00000100:00100000:3.0:1788201213.731643:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90235268b800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+6:13449:x1875064458210560:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201213.731653:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000002f75f507/0x4605c7c97e1becbf lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x2c8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b9b3e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.731693:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201213.731698:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90235268b800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+6:13449:x1875064458210560:12345-192.168.201.130@tcp:104: Request processed in 57us (2545us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201213.731703:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1299 00000100:00100000:3.0:1788201213.731843:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1300 00000100:00100000:3.0:1788201213.731851:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90235268a680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:13438:x1875064458210688:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201213.731861:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000002638eadf/0x4605c7c97e1c468b lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x40e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c9efb expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.731911:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1788201213.731917:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90235268a680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:13438:x1875064458210688:12345-192.168.201.130@tcp:104: Request processed in 67us (2530us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201213.731922:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1300 00000100:00100000:3.0:1788201213.731934:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1301 00000100:00100000:3.0:1788201213.731940:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023451b9c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13444:x1875064458210816:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201213.731949:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000028ee1549/0x4605c7c97e1c1f8d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x381:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c2e7d expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.731968:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to priority blp list, 3 blwis in pool 00000100:00100000:3.0:1788201213.731974:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023451b9c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13444:x1875064458210816:12345-192.168.201.130@tcp:104: Request processed in 34us (2567us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201213.731979:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1301 00010000:00010000:3.0:1788201213.732015:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:3.0:1788201213.732021:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000002f75f507/0x4605c7c97e1becbf lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x2c8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b9b3e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.732029:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234360a000 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201213.732034:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000002f75f507/0x4605c7c97e1becbf lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x2c8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4b9b3e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.732098:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000002f75f507/0x4605c7c97e1becbf lrc: 2/0,0 mode: --/PR res: [0x280000400:0x2c8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b9b3e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.732106:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201213.732125:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000002f75f507/0x4605c7c97e1becbf lrc: 1/0,0 mode: --/PR res: [0x280000400:0x2c8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b9b3e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.732133:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000002f75f507/0x4605c7c97e1becbf lrc: 0/0,0 mode: --/PR res: [0x280000400:0x2c8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b9b3e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.732150:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1788201213.732154:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000002638eadf/0x4605c7c97e1c468b lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x40e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c9efb expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201213.732500:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442608768:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201213.732831:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902353113600 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201213.732836:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000002638eadf/0x4605c7c97e1c468b lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x40e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4c9efb expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.732917:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000002638eadf/0x4605c7c97e1c468b lrc: 2/0,0 mode: --/PR res: [0x280000400:0x40e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c9efb expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.732924:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201213.732933:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000002638eadf/0x4605c7c97e1c468b lrc: 1/0,0 mode: --/PR res: [0x280000400:0x40e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c9efb expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.732940:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000002638eadf/0x4605c7c97e1c468b lrc: 0/0,0 mode: --/PR res: [0x280000400:0x40e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c9efb expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.732953:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201213.732955:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201213.732957:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201213.732968:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442609024:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201213.746068:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442608256:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201213.746077:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442608128:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201213.748083:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201213.748123:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201213.748150:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458211328 00000100:00100000:3.0:1788201213.748168:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458211328 00000100:00100000:3.0:1788201213.748206:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458211456 00000100:00100000:3.0:1788201213.748209:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458211456 00000100:00100000:3.0:1788201213.748222:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1302 00000100:00100000:3.0:1788201213.748231:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90235268bb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13441:x1875064458211328:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201213.748249:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000049aadd56/0x4605c7c97e1bff65 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x30c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4bd177 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.748317:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201213.748324:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90235268bb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13441:x1875064458211328:12345-192.168.201.130@tcp:104: Request processed in 94us (250us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201213.748343:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1302 00000100:00100000:3.0:1788201213.748360:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1303 00000100:00100000:3.0:1788201213.748366:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902352688380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13450:x1875064458211456:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201213.748374:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000006c367788/0x4605c7c97e1c419f lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x3fc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c90b5 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.748402:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1788201213.748407:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902352688380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13450:x1875064458211456:12345-192.168.201.130@tcp:104: Request processed in 42us (285us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201213.748413:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1303 00010000:00010000:3.0:1788201213.748535:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1788201213.748542:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000049aadd56/0x4605c7c97e1bff65 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x30c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4bd177 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.748551:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351efda00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201213.748556:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000049aadd56/0x4605c7c97e1bff65 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x30c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4bd177 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.748634:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000049aadd56/0x4605c7c97e1bff65 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x30c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bd177 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.748642:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201213.748660:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000049aadd56/0x4605c7c97e1bff65 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x30c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bd177 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.748668:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000049aadd56/0x4605c7c97e1bff65 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x30c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bd177 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.748689:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201213.748693:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000006c367788/0x4605c7c97e1c419f lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x3fc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c90b5 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.748699:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351f59000 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201213.748703:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000006c367788/0x4605c7c97e1c419f lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x3fc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4c90b5 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.748751:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000006c367788/0x4605c7c97e1c419f lrc: 2/0,0 mode: --/PR res: [0x280000400:0x3fc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c90b5 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.748759:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201213.748778:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000006c367788/0x4605c7c97e1c419f lrc: 1/0,0 mode: --/PR res: [0x280000400:0x3fc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c90b5 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.748786:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000006c367788/0x4605c7c97e1c419f lrc: 0/0,0 mode: --/PR res: [0x280000400:0x3fc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c90b5 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.748799:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201213.748801:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201213.748802:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201213.748808:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201213.748824:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442609280:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201213.748906:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201213.748912:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442609152:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201213.760313:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201213.760939:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201213.761723:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201213.761841:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458211712 00000100:00100000:3.0:1788201213.761849:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458211712 00000100:00100000:3.0:1788201213.761941:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458212096 00000100:00100000:3.0:1788201213.761945:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458212096 00000100:00100000:3.0:1788201213.761963:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1304 00000100:00100000:3.0:1788201213.761971:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c5180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:12363:x1875064458211712:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201213.761987:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004603efcb/0x4605c7c97e1bc352 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x232:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b23a4 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.762092:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201213.762099:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c5180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:12363:x1875064458211712:12345-192.168.201.130@tcp:104: Request processed in 128us (1797us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201213.762106:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1304 00000100:00100000:3.0:1788201213.762123:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1305 00000100:00100000:3.0:1788201213.762129:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c7b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13440:x1875064458212096:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201213.762137:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000005a9b077c/0x4605c7c97e1c038d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x31b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4bdd47 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201213.762312:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201213.762407:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201213.762462:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442608768:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201213.762495:0:11886:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1788201213.762506:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458212224 00000100:00100000:2.0:1788201213.762509:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458212224 00000100:00100000:2.0:1788201213.762522:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458212608 00000100:00100000:2.0:1788201213.762524:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458212608 00000100:00100000:2.0:1788201213.762531:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458212736 00000100:00100000:2.0:1788201213.762533:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458212736 00000100:00100000:2.0:1788201213.762542:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1306 00000100:00100000:2.0:1788201213.762548:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c5c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+7:13442:x1875064458212224:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201213.762556:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000006c2901fa/0x4605c7c97e1bd09c lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x262:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b49b4 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201213.766950:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1788201213.766967:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c7b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+7:13440:x1875064458212096:12345-192.168.201.130@tcp:104: Request processed in 4830us (6022us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201213.766980:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1305 00000100:00100000:3.0:1788201213.767024:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1307 00000100:00100000:3.0:1788201213.767038:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c6a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+6:13447:x1875064458212608:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201213.767054:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000f54f742f/0x4605c7c97e1c02bb lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x318:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4bdae6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.767136:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to priority blp list, 3 blwis in pool 00000100:00100000:3.0:1788201213.767141:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c6a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+6:13447:x1875064458212608:12345-192.168.201.130@tcp:104: Request processed in 111us (4831us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201213.767147:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1307 00000100:00100000:3.0:1788201213.767237:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1308 00000100:00100000:3.0:1788201213.767246:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902342df9050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13446:x1875064458212736:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201213.767255:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000fabf530c/0x4605c7c97e1c2f85 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x3bb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c5cac expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.767282:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/4 locks to priority blp list, 4 blwis in pool 00000100:00100000:3.0:1788201213.767287:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902342df9050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13446:x1875064458212736:12345-192.168.201.130@tcp:104: Request processed in 43us (4881us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201213.767292:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1308 00010000:00010000:3.0:1788201213.767308:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 3 total in blp. (3 blwis in pool) 00010000:00010000:3.0:1788201213.767313:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000005a9b077c/0x4605c7c97e1c038d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x31b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4bdd47 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.767321:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902353057c00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201213.767335:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000005a9b077c/0x4605c7c97e1c038d lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x31b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4bdd47 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.767410:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000005a9b077c/0x4605c7c97e1c038d lrc: 2/0,0 mode: --/PR res: [0x280000400:0x31b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bdd47 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.767418:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201213.767443:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000005a9b077c/0x4605c7c97e1c038d lrc: 1/0,0 mode: --/PR res: [0x280000400:0x31b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bdd47 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.767450:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000005a9b077c/0x4605c7c97e1c038d lrc: 0/0,0 mode: --/PR res: [0x280000400:0x31b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bdd47 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.767471:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:3.0:1788201213.767475:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000006c2901fa/0x4605c7c97e1bd09c lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x262:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4b49b4 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201213.767482:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352f7a200 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201213.767486:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000006c2901fa/0x4605c7c97e1bd09c lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x262:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4b49b4 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201213.767521:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000006c2901fa/0x4605c7c97e1bd09c lrc: 2/0,0 mode: --/PR res: [0x280000400:0x262:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b49b4 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201213.767529:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201213.767535:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000006c2901fa/0x4605c7c97e1bd09c lrc: 1/0,0 mode: --/PR res: [0x280000400:0x262:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b49b4 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201213.767542:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000006c2901fa/0x4605c7c97e1bd09c lrc: 0/0,0 mode: --/PR res: [0x280000400:0x262:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4b49b4 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201213.767555:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1788201213.767559:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000f54f742f/0x4605c7c97e1c02bb lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x318:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4bdae6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.767570:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9023483d1e00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201213.767574:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000f54f742f/0x4605c7c97e1c02bb lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x318:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4bdae6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.767605:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000f54f742f/0x4605c7c97e1c02bb lrc: 2/0,0 mode: --/PR res: [0x280000400:0x318:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bdae6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.767612:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201213.767617:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000f54f742f/0x4605c7c97e1c02bb lrc: 1/0,0 mode: --/PR res: [0x280000400:0x318:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bdae6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.767623:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000f54f742f/0x4605c7c97e1c02bb lrc: 0/0,0 mode: --/PR res: [0x280000400:0x318:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4bdae6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.767637:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201213.767640:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000fabf530c/0x4605c7c97e1c2f85 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x3bb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4c5cac expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.767647:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9023436c4200 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201213.767650:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000fabf530c/0x4605c7c97e1c2f85 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x3bb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4c5cac expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.767679:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000fabf530c/0x4605c7c97e1c2f85 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x3bb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c5cac expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.767692:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201213.767697:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000fabf530c/0x4605c7c97e1c2f85 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x3bb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c5cac expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.767704:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000fabf530c/0x4605c7c97e1c2f85 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x3bb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4c5cac expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201213.767853:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201213.767854:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201213.767856:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201213.767900:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442609536:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201213.767956:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442609792:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201213.767998:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442609152:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201213.768009:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442609280:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201213.768019:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442609024:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201213.768042:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201213.768047:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442609408:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201213.768068:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442609664:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201213.768269:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/0 locks to priority blp list, 0 blwis in pool 00000100:00100000:2.0:1788201213.768278:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c5c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13442:x1875064458212224:12345-192.168.201.130@tcp:104: Request processed in 5729us (6555us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201213.768286:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1306 00010000:00010000:2.0:1788201213.768469:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201213.769011:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201213.774757:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442609408:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201213.774850:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442609664:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201213.774860:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442609536:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201213.774869:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442609792:192.168.201.130@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1788201213.906865:0:13659:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201213.907274:0:13659:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201213.907350:0:13659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c49d3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 13659 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201213.907362:0:13659:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c49d3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 13659 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201213.907372:0:13659:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c49d3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 13659 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201213.907400:0:13659:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13659:1875064442610048:192.168.201.130@tcp:101:lfs.0 00000100:00100000:2.0:1788201213.907540:0:13659:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201213.916025:0:13659:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13659:1875064442610048:192.168.201.130@tcp:101:lfs.0 00010000:00010000:3.0:1788201213.916064:0:13659:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c49d3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2e97 expref: -99 pid: 13659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201213.916082:0:13659:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c49d3 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2e97 expref: -99 pid: 13659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201213.916098:0:13659:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c49d3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2e97 expref: -99 pid: 13659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201213.916111:0:13659:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c49d3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2e97 expref: -99 pid: 13659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201213.916127:0:13659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c49d3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2e97 expref: -99 pid: 13659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201213.916137:0:13659:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c49d3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2e97 expref: -99 pid: 13659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201213.916146:0:13659:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c49d3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2e97 expref: -99 pid: 13659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201213.916162:0:13659:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c5180 x1875064442610048/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201229 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201213.916181:0:13659:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c49d3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2e97 expref: -99 pid: 13659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201213.916197:0:13659:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201213.916206:0:13659:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201213.916258:0:13659:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:3.0:1788201213.916266:0:13659:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c49d3 00000080:00010000:3.0:1788201213.916274:0:13659:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201213.916278:0:13659:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49d3 from it ffffa0b181d4bb58 00010000:00010000:3.0:1788201213.916282:0:13659:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c49d3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2e97 expref: -99 pid: 13659 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201213.916292:0:13659:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c49d3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2e97 expref: -99 pid: 13659 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201213.917067:0:13659:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201213.917259:0:13659:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d102a.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=13 00000080:00200000:2.0:1788201213.917274:0:13659:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d102a.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=getattr 00000002:00010000:2.0:1788201213.917312:0:13659:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d102a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201213.917403:0:13659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c49da lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 13659 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201213.917415:0:13659:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c49da lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13659 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201213.917428:0:13659:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c49da lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13659 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201213.917463:0:13659:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13659:1875064442610176:192.168.201.130@tcp:101:lfs.0 00000100:00100000:2.0:1788201213.917586:0:13659:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448f00 going to sleep for 11 seconds 00000080:00200080:2.0:1788201213.933727:0:13659:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c03066f0 arg=7fffd9bba820 00000080:00200000:2.0:1788201213.933740:0:13659:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) name=d102a.replay-single stripe_offset=-1 stripe_count=1, hash_type=0 00800000:00000004:2.0:1788201213.933770:0:13659:0:(obd_class.h:1057:obd_statfs_async()) lustre-MDT0000-mdc-ffff902350a11000: age 471, max_age 0 00000100:00100000:2.0:1788201213.933832:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442610304:192.168.201.130@tcp:41:lfs.0 00010000:00010000:2.0:1788201213.933965:0:13659:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c49d3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d2e97 expref: -99 pid: 13659 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788201213.934009:0:13659:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c49d3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d2e97 expref: -99 pid: 13659 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788201213.934017:0:13659:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201213.934020:0:13659:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c49d3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d2e97 expref: -99 pid: 13659 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000020:00100000:2.0:1788201213.934042:0:13659:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201213.934049:0:13659:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c5c00 x1875064442610560/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 568/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201213.934065:0:13659:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5c00 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13659:1875064442610560:192.168.201.130@tcp:36:lfs.0 00000100:00100000:2.0:1788201213.934086:0:13659:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448240 going to sleep for 11 seconds 00000100:00100000:3.0:1788201213.942276:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442610304:192.168.201.130@tcp:41:lfs.0 00000080:00200000:2.0:1788201213.968255:0:13662:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201213.968283:0:13662:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201213.968358:0:13662:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 13662 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201213.968370:0:13662:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 13662 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201213.968381:0:13662:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 13662 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201213.968404:0:13662:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13662:1875064442610688:192.168.201.130@tcp:101:lfs.0 00000100:00100000:2.0:1788201213.968741:0:13662:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448e40 going to sleep for 11 seconds 00000080:00200000:2.0:1788201214.103497:0:13665:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201214.103535:0:13665:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201214.103624:0:13665:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201214.103642:0:13665:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201214.103651:0:13665:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201214.103659:0:13665:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201214.103663:0:13665:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c49e1 00000080:00010000:2.0:1788201214.103671:0:13665:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49e1 from it ffffa0b180d4bbf8 00010000:00010000:2.0:1788201214.103677:0:13665:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201214.103687:0:13665:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201214.103704:0:13665:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201214.103717:0:13665:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d102a.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=25 00000080:00200000:2.0:1788201214.103721:0:13665:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d102a.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=0 00000002:00010000:2.0:1788201214.103731:0:13665:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d102a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1788201214.103986:0:13665:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c49f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 13665 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.103995:0:13665:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c49f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13665 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.104005:0:13665:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c49f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13665 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201214.104029:0:13665:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7100 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13665:1875064442611968:192.168.201.130@tcp:101:touch.0 00000100:00100000:3.0:1788201214.104291:0:13665:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448300 going to sleep for 11 seconds 00000100:00100000:2.0:1788201214.107793:0:13665:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7100 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13665:1875064442611968:192.168.201.130@tcp:101:touch.0 00010000:00010000:2.0:1788201214.107843:0:13665:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c49f6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2f23 expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.107866:0:13665:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c49f6 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2f23 expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.107909:0:13665:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c49f6 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2f23 expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.107924:0:13665:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c49f6 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2f23 expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.107940:0:13665:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c49f6 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2f23 expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.107958:0:13665:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c49f6 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2f23 expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.107968:0:13665:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c49f6 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2f23 expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.107977:0:13665:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c49f6 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2f23 expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201214.107991:0:13665:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9023458c7100 x1875064442611968/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201230 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201214.108007:0:13665:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c49f6 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2f23 expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.108019:0:13665:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1001 (0 0) 00000080:00200000:2.0:1788201214.108028:0:13665:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201214.108056:0:13665:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x3e9:0x0], updating i_size 512 00000080:00200000:2.0:1788201214.108060:0:13665:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3e9:0x0](ffff9023449b1148): rc = 0 00000080:00010000:2.0:1788201214.108067:0:13665:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3e9:0x0](ffff9023449b1148) for lock 0x4605c7c97e1c49f6 00000080:00010000:2.0:1788201214.108082:0:13665:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3e9:0x0](ffff9023449b1148) 00000080:00010000:2.0:1788201214.108087:0:13665:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49f6 from it ffffa0b180d4bac8 00010000:00010000:2.0:1788201214.108091:0:13665:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c49f6 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2f23 expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.108102:0:13665:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c49f6 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2f23 expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201214.108143:0:13665:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201214.108167:0:13665:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 3 00000080:00200000:2.0:1788201214.108174:0:13665:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file-1, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), file ffff9023460b9080, open_flags 8941, mode 81b6 opened 0 00000080:00200000:2.0:1788201214.108184:0:13665:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-1, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), intent=open|creat 00000002:00010000:2.0:1788201214.108213:0:13665:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-1,[0x200000bd1:0x3ea:0x0]) in obj [0x200000bd1:0x3e9:0x0], intent: open|creat flags 0104502 00010000:00010000:2.0:1788201214.108226:0:13665:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c49ef lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d2f00 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201214.108786:0:13665:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c49ef lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d2f00 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201214.108805:0:13665:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201214.108809:0:13665:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c49ef lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d2f00 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201214.108843:0:13665:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 13665 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.108850:0:13665:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13665 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201214.108865:0:13665:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201214.108904:0:13665:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c7100 x1875064442612224/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201214.108915:0:13665:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13665 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201214.108943:0:13665:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7100 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13665:1875064442612224:192.168.201.130@tcp:101:touch.0 00000100:00100000:2.0:1788201214.109151:0:13665:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448300 going to sleep for 11 seconds 00000100:00100000:2.0:1788201214.117535:0:13665:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7100 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13665:1875064442612224:192.168.201.130@tcp:101:touch.0 00010000:00010000:2.0:1788201214.117575:0:13665:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2f3f expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.117609:0:13665:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x3ea:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2f3f expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.117619:0:13665:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x3ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2f3f expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.117630:0:13665:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x3ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2f3f expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.117643:0:13665:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x3ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2f3f expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.117658:0:13665:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x3ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2f3f expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.117668:0:13665:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x3ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2f3f expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.117676:0:13665:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2f3f expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201214.117690:0:13665:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023458c7100 x1875064442612224/t8589938612(8589938612) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201230 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201214.117706:0:13665:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458c7100 x1875064442612224/t8589938612(8589938612) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201230 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201214.117910:0:13665:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2f3f expref: -99 pid: 13665 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201214.117928:0:13665:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2f3f expref: -99 pid: 13665 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.117941:0:13665:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1002 (0 0) 00000080:00200000:2.0:1788201214.117949:0:13665:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x3e9:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201214.117966:0:13665:0:(namei.c:1628:ll_create_it()) VFS Op:name=file-1, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), intent=open|creat 00000080:00200000:2.0:1788201214.118007:0:13665:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x3ea:0x0](ffff9023449a9148) 00000080:00200000:2.0:1788201214.118170:0:13665:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3ea:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788201214.118180:0:13665:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x3ea:0x0](ffff9023449a9148) 00000080:00010000:2.0:1788201214.118184:0:13665:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3ea:0x0](ffff9023449a9148) for lock 0x4605c7c97e1c49fd 00000080:00010000:2.0:1788201214.118195:0:13665:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3ea:0x0](ffff9023449a9148) 00000080:00200008:2.0:1788201214.118515:0:13665:0:(file.c:1090:ll_file_open()) START file file-1:[0x200000bd1:0x3ea:0x0](ffff9023449a9148), flags 104501 00000002:00100000:2.0:1788201214.118542:0:13665:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458c7100 x1875064442612224/t8589938612(8589938612) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201230 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788201214.118591:0:13665:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49fd from it ffff902342d498a0 00010000:00010000:2.0:1788201214.118597:0:13665:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d2f3f expref: -99 pid: 13665 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.118608:0:13665:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d2f3f expref: -99 pid: 13665 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201214.118636:0:13665:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3ea:0x0](ffff9023449a9148), inode mode 81a4 mask 40 00000080:00200008:2.0:1788201214.118683:0:13665:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-1:[0x200000bd1:0x3ea:0x0](ffff9023449a9148) current size 0, valid attrs 20070, mode 9023, uid -2133541296, gid -24399, new size 18446744071973566999, atime 1788201214.000000000, mtime 1788201214.000000000, ctime 1788201214.000000000 00010000:00010000:2.0:1788201214.118709:0:13665:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d2f3f expref: -99 pid: 13665 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201214.119037:0:13665:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3ea:0x0]: losing layout lock 00010000:00010000:2.0:1788201214.119102:0:13665:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d2f3f expref: -99 pid: 13665 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201214.119110:0:13665:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201214.119113:0:13665:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c49fd lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d2f3f expref: -99 pid: 13665 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788201214.119148:0:13665:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201214.119158:0:13665:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c4700 x1875064442612480/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201214.119189:0:13665:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13665:1875064442612480:192.168.201.130@tcp:36:touch.0 00000100:00100000:2.0:1788201214.119629:0:13665:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448840 going to sleep for 11 seconds 00000080:00200000:2.0:1788201214.133595:0:13665:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3ea:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201214.133627:0:13665:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-1:[0x200000bd1:0x3ea:0x0](ffff9023449a9148) current size 0, valid attrs 100201f0, mode 9023, uid -2133541296, gid -24399, new size 18446744071973566999, atime 1788201214.000000000, mtime 1788201214.000000000, ctime 1788201214.000000000, rc 0 00000080:00200008:2.0:1788201214.133667:0:13665:0:(file.c:479:ll_file_release()) START file file-1:[0x200000bd1:0x3ea:0x0](ffff9023449a9148), flags 104001 00000002:00100000:2.0:1788201214.133755:0:13665:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023458c7100 x1875064442612224/t8589938612(8589938612) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201230 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201214.133770:0:13665:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201214.133777:0:13665:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c4380 x1875064442612992/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201214.133822:0:13665:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4380 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13665:1875064442612992:192.168.201.130@tcp:35:touch.0 00000100:00100000:2.0:1788201214.133968:0:13665:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448b40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201214.137290:0:13665:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4380 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13665:1875064442612992:192.168.201.130@tcp:35:touch.0 00000002:00100000:2.0:1788201214.137357:0:13665:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023458c7100 x1875064442612224/t8589938612(8589938612) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201230 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201214.148080:0:13666:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201214.148122:0:13666:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201214.148149:0:13666:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201214.148173:0:13666:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201214.148182:0:13666:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201214.148192:0:13666:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201214.148196:0:13666:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c49e1 00000080:00010000:2.0:1788201214.148285:0:13666:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49e1 from it ffffa0b181c5bbf8 00010000:00010000:2.0:1788201214.148290:0:13666:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201214.148380:0:13666:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201214.148402:0:13666:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201214.148411:0:13666:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102a.replay-single, flags=25 00000080:00200000:2.0:1788201214.148418:0:13666:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201214.148581:0:13666:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 3 00000080:00200000:2.0:1788201214.148589:0:13666:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file-2, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), file ffff902344d6e480, open_flags 8941, mode 81b6 opened 0 00000080:00200000:2.0:1788201214.148599:0:13666:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-2, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), intent=open|creat 00000002:00010000:2.0:1788201214.148624:0:13666:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-2,[0x200000bd1:0x3eb:0x0]) in obj [0x200000bd1:0x3e9:0x0], intent: open|creat flags 0104502 00010000:00010000:2.0:1788201214.148695:0:13666:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c4a0b lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 13666 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.148705:0:13666:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c4a0b lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13666 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201214.148721:0:13666:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201214.148739:0:13666:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c6680 x1875064442613248/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201214.148751:0:13666:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c4a0b lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13666 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201214.148781:0:13666:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13666:1875064442613248:192.168.201.130@tcp:101:touch.0 00000100:00100000:2.0:1788201214.149630:0:13666:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448b40 going to sleep for 11 seconds 00000080:00200000:2.0:1788201214.252479:0:13667:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3ec:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:2.0:1788201214.252510:0:13667:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x3ec:0x0](ffff9023449a3248) 00000002:00010000:2.0:1788201214.252525:0:13667:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3ec:0x0]) in obj [0x200000bd1:0x3ec:0x0], intent: layout flags 0 00010000:00010000:2.0:1788201214.252598:0:13667:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4a20 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3ec:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 13667 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.252609:0:13667:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4a20 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3ec:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 13667 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.252620:0:13667:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4a20 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3ec:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 13667 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201214.252644:0:13667:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6d80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13667:1875064442614656:192.168.201.130@tcp:101:touch.0 00000100:00100000:2.0:1788201214.253019:0:13667:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448900 going to sleep for 11 seconds 00000080:00200000:2.0:1788201214.262390:0:13667:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3ec:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201214.262495:0:13667:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-3:[0x200000bd1:0x3ec:0x0](ffff9023449a3248) current size 0, valid attrs 100201f0, mode 9023, uid -2117747120, gid -24399, new size 18446744071973566999, atime 1788201214.000000000, mtime 1788201214.000000000, ctime 1788201214.000000000, rc 0 00000080:00200008:2.0:1788201214.262536:0:13667:0:(file.c:479:ll_file_release()) START file file-3:[0x200000bd1:0x3ec:0x0](ffff9023449a3248), flags 104001 00000002:00100000:2.0:1788201214.262620:0:13667:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff902345833100 x1875064442614272/t8589938618(8589938618) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201230 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201214.262634:0:13667:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201214.262639:0:13667:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c6d80 x1875064442615040/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201214.262667:0:13667:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6d80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13667:1875064442615040:192.168.201.130@tcp:35:touch.0 00000100:00100000:2.0:1788201214.263065:0:13667:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448180 going to sleep for 11 seconds 00000100:00100000:2.0:1788201214.266730:0:13667:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6d80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13667:1875064442615040:192.168.201.130@tcp:35:touch.0 00000002:00100000:2.0:1788201214.266774:0:13667:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff902345833100 x1875064442614272/t8589938618(8589938618) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201230 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201214.305492:0:13668:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3ed:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201214.305523:0:13668:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-4:[0x200000bd1:0x3ed:0x0](ffff9023449b00c8) current size 0, valid attrs 100201f0, mode 9023, uid -2125316528, gid -24399, new size 18446744071973566999, atime 1788201214.000000000, mtime 1788201214.000000000, ctime 1788201214.000000000, rc 0 00000080:00200008:2.0:1788201214.305561:0:13668:0:(file.c:479:ll_file_release()) START file file-4:[0x200000bd1:0x3ed:0x0](ffff9023449b00c8), flags 104001 00000002:00100000:2.0:1788201214.305633:0:13668:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b0380 x1875064442615296/t8589938621(8589938621) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201230 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201214.305646:0:13668:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201214.305652:0:13668:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c7800 x1875064442616064/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201214.305678:0:13668:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13668:1875064442616064:192.168.201.130@tcp:35:touch.0 00000100:00100000:2.0:1788201214.305857:0:13668:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448180 going to sleep for 11 seconds 00000100:00100000:2.0:1788201214.308655:0:13668:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13668:1875064442616064:192.168.201.130@tcp:35:touch.0 00000002:00100000:2.0:1788201214.308695:0:13668:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b0380 x1875064442615296/t8589938621(8589938621) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201230 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201214.317145:0:13669:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201214.317188:0:13669:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201214.317279:0:13669:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788201214.317367:0:13669:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:2.0:1788201214.317379:0:13669:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201214.317390:0:13669:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201214.317395:0:13669:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c49e1 00000080:00010000:2.0:1788201214.317404:0:13669:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49e1 from it ffffa0b181d4bbf8 00010000:00010000:2.0:1788201214.317409:0:13669:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788201214.317417:0:13669:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788201214.317497:0:13669:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201214.317508:0:13669:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102a.replay-single, flags=25 00000080:00200000:2.0:1788201214.317514:0:13669:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201214.317529:0:13669:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 3 00000080:00200000:2.0:1788201214.317534:0:13669:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file-5, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), file ffff902344d6f380, open_flags 8941, mode 81b6 opened 0 00000080:00200000:2.0:1788201214.317544:0:13669:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-5, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), intent=open|creat 00000002:00010000:2.0:1788201214.317570:0:13669:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-5,[0x200000bd1:0x3ee:0x0]) in obj [0x200000bd1:0x3e9:0x0], intent: open|creat flags 0104502 00010000:00010000:2.0:1788201214.317944:0:13669:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 13669 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.317954:0:13669:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13669 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201214.317971:0:13669:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201214.317987:0:13669:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c6a00 x1875064442616320/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201214.317998:0:13669:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13669 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201214.318027:0:13669:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13669:1875064442616320:192.168.201.130@tcp:101:touch.0 00000100:00100000:2.0:1788201214.318253:0:13669:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448180 going to sleep for 11 seconds 00000100:00100000:2.0:1788201214.328512:0:13669:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13669:1875064442616320:192.168.201.130@tcp:101:touch.0 00010000:00010000:2.0:1788201214.328552:0:13669:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3003 expref: -99 pid: 13669 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.328582:0:13669:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3003 expref: -99 pid: 13669 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.328592:0:13669:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3003 expref: -99 pid: 13669 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.328605:0:13669:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3003 expref: -99 pid: 13669 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.328619:0:13669:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3003 expref: -99 pid: 13669 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.328634:0:13669:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3003 expref: -99 pid: 13669 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.328643:0:13669:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3003 expref: -99 pid: 13669 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.328651:0:13669:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3003 expref: -99 pid: 13669 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201214.328667:0:13669:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023458c6a00 x1875064442616320/t8589938624(8589938624) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201230 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201214.328683:0:13669:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458c6a00 x1875064442616320/t8589938624(8589938624) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201230 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201214.328699:0:13669:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3003 expref: -99 pid: 13669 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201214.328713:0:13669:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3003 expref: -99 pid: 13669 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.328725:0:13669:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1006 (0 0) 00000080:00200000:2.0:1788201214.328733:0:13669:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x3e9:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201214.328750:0:13669:0:(namei.c:1628:ll_create_it()) VFS Op:name=file-5, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), intent=open|creat 00000080:00200000:2.0:1788201214.328786:0:13669:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x3ee:0x0](ffff9023449aa1c8) 00000080:00200000:2.0:1788201214.328973:0:13669:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3ee:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788201214.328986:0:13669:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x3ee:0x0](ffff9023449aa1c8) 00000080:00010000:2.0:1788201214.328991:0:13669:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3ee:0x0](ffff9023449aa1c8) for lock 0x4605c7c97e1c4a35 00000080:00010000:2.0:1788201214.329003:0:13669:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3ee:0x0](ffff9023449aa1c8) 00000080:00200008:2.0:1788201214.329022:0:13669:0:(file.c:1090:ll_file_open()) START file file-5:[0x200000bd1:0x3ee:0x0](ffff9023449aa1c8), flags 104501 00000002:00100000:2.0:1788201214.329047:0:13669:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458c6a00 x1875064442616320/t8589938624(8589938624) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201230 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788201214.329093:0:13669:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4a35 from it ffff902342d49120 00010000:00010000:2.0:1788201214.329099:0:13669:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3003 expref: -99 pid: 13669 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.329109:0:13669:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3003 expref: -99 pid: 13669 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201214.329141:0:13669:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3ee:0x0](ffff9023449aa1c8), inode mode 81a4 mask 40 00000080:00200008:2.0:1788201214.329188:0:13669:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-5:[0x200000bd1:0x3ee:0x0](ffff9023449aa1c8) current size 0, valid attrs 20070, mode 9023, uid -2116764080, gid -24399, new size 18446744071973566999, atime 1788201214.000000000, mtime 1788201214.000000000, ctime 1788201214.000000000 00010000:00010000:2.0:1788201214.329213:0:13669:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d3003 expref: -99 pid: 13669 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201214.329240:0:13669:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3ee:0x0]: losing layout lock 00010000:00010000:2.0:1788201214.329462:0:13669:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3003 expref: -99 pid: 13669 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201214.329471:0:13669:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201214.329474:0:13669:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a35 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3003 expref: -99 pid: 13669 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788201214.329508:0:13669:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201214.329517:0:13669:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c7b80 x1875064442616576/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201214.329545:0:13669:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13669:1875064442616576:192.168.201.130@tcp:36:touch.0 00000100:00100000:2.0:1788201214.329634:0:13669:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201214.336063:0:13669:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13669:1875064442616576:192.168.201.130@tcp:36:touch.0 00000080:00200000:2.0:1788201214.336171:0:13669:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3ee:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:2.0:1788201214.336550:0:13669:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x3ee:0x0](ffff9023449aa1c8) 00000002:00010000:2.0:1788201214.336566:0:13669:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3ee:0x0]) in obj [0x200000bd1:0x3ee:0x0], intent: layout flags 0 00010000:00010000:2.0:1788201214.336632:0:13669:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a3c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 13669 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.336648:0:13669:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a3c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 13669 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.336659:0:13669:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a3c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 13669 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201214.336684:0:13669:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13669:1875064442616704:192.168.201.130@tcp:101:touch.0 00000100:00100000:2.0:1788201214.339552:0:13669:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201214.344622:0:13669:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13669:1875064442616704:192.168.201.130@tcp:101:touch.0 00010000:00010000:3.0:1788201214.344660:0:13669:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a3c lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3018 expref: -99 pid: 13669 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201214.344677:0:13669:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a3c lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3018 expref: -99 pid: 13669 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201214.344693:0:13669:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9023458c5180 x1875064442616704/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201230 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201214.344706:0:13669:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a3c lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3018 expref: -99 pid: 13669 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201214.345045:0:13669:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a3c lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3018 expref: -99 pid: 13669 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201214.345066:0:13669:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1006 (0 0) 00000080:00200000:3.0:1788201214.345074:0:13669:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x3ee:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1788201214.345095:0:13669:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3ee:0x0](ffff9023449aa1c8) for lock 0x4605c7c97e1c4a3c 00000080:00010000:3.0:1788201214.345105:0:13669:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4a3c from it ffffa0b181d4ba78 00010000:00010000:3.0:1788201214.345110:0:13669:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a3c lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3018 expref: -99 pid: 13669 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201214.345121:0:13669:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a3c lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3018 expref: -99 pid: 13669 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201214.345149:0:13669:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a3c lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3018 expref: -99 pid: 13669 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201214.345161:0:13669:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a3c lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3018 expref: -99 pid: 13669 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201214.345170:0:13669:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x3ee:0x0](0000000097d62046) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a3c lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3018 expref: -99 pid: 13669 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201214.345260:0:13669:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3ee:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:3.0:1788201214.345283:0:13669:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a3c lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3018 expref: -99 pid: 13669 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201214.345299:0:13669:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4a3c lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3018 expref: -99 pid: 13669 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:3.0:1788201214.345329:0:13669:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201214.345439:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201214.345461:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442616832:192.168.201.130@tcp:2:touch.0 00000100:00100000:3.0:1788201214.348973:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442616832:192.168.201.130@tcp:2:touch.0 00000080:00200000:2.0:1788201214.349263:0:13669:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3ee:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201214.349306:0:13669:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-5:[0x200000bd1:0x3ee:0x0](ffff9023449aa1c8) current size 0, valid attrs 100201f0, mode 9023, uid -2116764080, gid -24399, new size 18446744071973566999, atime 1788201214.000000000, mtime 1788201214.000000000, ctime 1788201214.000000000, rc 0 00000080:00200008:2.0:1788201214.349354:0:13669:0:(file.c:479:ll_file_release()) START file file-5:[0x200000bd1:0x3ee:0x0](ffff9023449aa1c8), flags 104001 00000002:00100000:2.0:1788201214.349449:0:13669:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023458c6a00 x1875064442616320/t8589938624(8589938624) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201230 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201214.349466:0:13669:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201214.349472:0:13669:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c6300 x1875064442617088/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201214.349498:0:13669:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6300 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13669:1875064442617088:192.168.201.130@tcp:35:touch.0 00000100:00100000:2.0:1788201214.349583:0:13669:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448780 going to sleep for 11 seconds 00000100:00100000:3.0:1788201214.353254:0:13669:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6300 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13669:1875064442617088:192.168.201.130@tcp:35:touch.0 00000002:00100000:3.0:1788201214.353394:0:13669:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023458c6a00 x1875064442616320/t8589938624(8589938624) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201230 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201214.360599:0:13671:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201214.360635:0:13671:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201214.360659:0:13671:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788201214.360675:0:13671:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:3.0:1788201214.360682:0:13671:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201214.360691:0:13671:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201214.360694:0:13671:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c49e1 00000080:00010000:3.0:1788201214.360702:0:13671:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49e1 from it ffffa0b181bdbbf8 00010000:00010000:3.0:1788201214.360705:0:13671:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788201214.360712:0:13671:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:3.0:1788201214.360728:0:13671:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201214.360735:0:13671:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102a.replay-single, flags=25 00000080:00200000:3.0:1788201214.360740:0:13671:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 1 00000080:00200000:3.0:1788201214.360749:0:13671:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 3 00000080:00200000:3.0:1788201214.360753:0:13671:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file-6, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), file ffff902342d8d380, open_flags 8941, mode 81b6 opened 0 00000080:00200000:3.0:1788201214.360761:0:13671:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-6, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), intent=open|creat 00000002:00010000:3.0:1788201214.360781:0:13671:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-6,[0x200000bd1:0x3ef:0x0]) in obj [0x200000bd1:0x3e9:0x0], intent: open|creat flags 0104502 00010000:00010000:3.0:1788201214.360848:0:13671:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 13671 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201214.360857:0:13671:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13671 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201214.360871:0:13671:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201214.361904:0:13671:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90235268a680 x1875064442617344/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201214.361920:0:13671:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13671 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201214.362172:0:13671:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13671:1875064442617344:192.168.201.130@tcp:101:touch.0 00000100:00100000:3.0:1788201214.362400:0:13671:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3780 going to sleep for 11 seconds 00000100:00100000:3.0:1788201214.366143:0:13671:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13671:1875064442617344:192.168.201.130@tcp:101:touch.0 00010000:00010000:3.0:1788201214.366175:0:13671:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3034 expref: -99 pid: 13671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201214.366195:0:13671:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3034 expref: -99 pid: 13671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201214.366202:0:13671:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3034 expref: -99 pid: 13671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201214.366223:0:13671:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3034 expref: -99 pid: 13671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201214.366232:0:13671:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3034 expref: -99 pid: 13671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201214.366244:0:13671:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3034 expref: -99 pid: 13671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201214.366250:0:13671:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3034 expref: -99 pid: 13671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201214.366254:0:13671:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3034 expref: -99 pid: 13671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201214.366262:0:13671:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff90235268a680 x1875064442617344/t8589938627(8589938627) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201230 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201214.366272:0:13671:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90235268a680 x1875064442617344/t8589938627(8589938627) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201230 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201214.366283:0:13671:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3034 expref: -99 pid: 13671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201214.366294:0:13671:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3034 expref: -99 pid: 13671 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201214.366306:0:13671:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1007 (0 0) 00000080:00200000:3.0:1788201214.366340:0:13671:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x3e9:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201214.366358:0:13671:0:(namei.c:1628:ll_create_it()) VFS Op:name=file-6, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), intent=open|creat 00000080:00200000:3.0:1788201214.366389:0:13671:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x3ef:0x0](ffff90234497db88) 00000080:00200000:2.0:1788201214.367496:0:13671:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3ef:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788201214.367517:0:13671:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x3ef:0x0](ffff90234497db88) 00000080:00010000:2.0:1788201214.367521:0:13671:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3ef:0x0](ffff90234497db88) for lock 0x4605c7c97e1c4a43 00000080:00010000:2.0:1788201214.367533:0:13671:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3ef:0x0](ffff90234497db88) 00000080:00200008:2.0:1788201214.367550:0:13671:0:(file.c:1090:ll_file_open()) START file file-6:[0x200000bd1:0x3ef:0x0](ffff90234497db88), flags 104501 00000002:00100000:2.0:1788201214.367575:0:13671:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90235268a680 x1875064442617344/t8589938627(8589938627) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201230 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788201214.367609:0:13671:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4a43 from it ffff9023533ea2a0 00010000:00010000:2.0:1788201214.367613:0:13671:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3034 expref: -99 pid: 13671 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.367622:0:13671:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3034 expref: -99 pid: 13671 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201214.367646:0:13671:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3ef:0x0](ffff90234497db88), inode mode 81a4 mask 40 00000080:00200008:2.0:1788201214.367687:0:13671:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-6:[0x200000bd1:0x3ef:0x0](ffff90234497db88) current size 0, valid attrs 20070, mode 9023, uid -2118271408, gid -24399, new size 18446744071973566999, atime 1788201214.000000000, mtime 1788201214.000000000, ctime 1788201214.000000000 00010000:00010000:2.0:1788201214.367707:0:13671:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d3034 expref: -99 pid: 13671 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201214.367772:0:13671:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3ef:0x0]: losing layout lock 00010000:00010000:2.0:1788201214.367869:0:13671:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3034 expref: -99 pid: 13671 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201214.367908:0:13671:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201214.367912:0:13671:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4a43 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3034 expref: -99 pid: 13671 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788201214.367947:0:13671:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201214.367954:0:13671:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c4e00 x1875064442617600/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201214.367982:0:13671:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4e00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13671:1875064442617600:192.168.201.130@tcp:36:touch.0 00000100:00100000:2.0:1788201214.368082:0:13671:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584489c0 going to sleep for 11 seconds 00000080:00200000:3.0:1788201214.382830:0:13672:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201214.382869:0:13672:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201214.382930:0:13672:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788201214.382952:0:13672:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:3.0:1788201214.382962:0:13672:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201214.382972:0:13672:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201214.382976:0:13672:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c49e1 00000080:00010000:3.0:1788201214.382987:0:13672:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49e1 from it ffffa0b180d4bbf8 00010000:00010000:3.0:1788201214.382992:0:13672:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788201214.383003:0:13672:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1788201214.383024:0:13672:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201214.383034:0:13672:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102a.replay-single, flags=25 00000080:00200000:3.0:1788201214.383041:0:13672:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 1 00000080:00200000:3.0:1788201214.383051:0:13672:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 3 00000080:00200000:3.0:1788201214.383055:0:13672:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file-7, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), file ffff902350898180, open_flags 8941, mode 81b6 opened 0 00000080:00200000:3.0:1788201214.383063:0:13672:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-7, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), intent=open|creat 00000002:00010000:3.0:1788201214.383088:0:13672:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-7,[0x200000bd1:0x3f0:0x0]) in obj [0x200000bd1:0x3e9:0x0], intent: open|creat flags 0104502 00010000:00010000:3.0:1788201214.383154:0:13672:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 13672 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201214.383161:0:13672:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13672 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201214.383175:0:13672:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201214.383189:0:13672:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90235268b800 x1875064442618368/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201214.383196:0:13672:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13672 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201214.383224:0:13672:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13672:1875064442618368:192.168.201.130@tcp:101:touch.0 00000100:00100000:3.0:1788201214.383530:0:13672:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3780 going to sleep for 11 seconds 00000100:00100000:2.0:1788201214.388146:0:13672:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13672:1875064442618368:192.168.201.130@tcp:101:touch.0 00010000:00010000:2.0:1788201214.388184:0:13672:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3065 expref: -99 pid: 13672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.388220:0:13672:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3065 expref: -99 pid: 13672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.388229:0:13672:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3065 expref: -99 pid: 13672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.388244:0:13672:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3065 expref: -99 pid: 13672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.388256:0:13672:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3065 expref: -99 pid: 13672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.388270:0:13672:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3065 expref: -99 pid: 13672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.388276:0:13672:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3065 expref: -99 pid: 13672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.388281:0:13672:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3065 expref: -99 pid: 13672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201214.388290:0:13672:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff90235268b800 x1875064442618368/t8589938630(8589938630) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201230 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201214.388307:0:13672:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90235268b800 x1875064442618368/t8589938630(8589938630) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201230 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201214.388332:0:13672:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3065 expref: -99 pid: 13672 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201214.388347:0:13672:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3065 expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.388359:0:13672:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1008 (0 0) 00000080:00200000:2.0:1788201214.388367:0:13672:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x3e9:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201214.388385:0:13672:0:(namei.c:1628:ll_create_it()) VFS Op:name=file-7, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), intent=open|creat 00000080:00200000:2.0:1788201214.388427:0:13672:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x3f0:0x0](ffff9023449a9988) 00000080:00200000:2.0:1788201214.388591:0:13672:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3f0:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788201214.388603:0:13672:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x3f0:0x0](ffff9023449a9988) 00000080:00010000:2.0:1788201214.388607:0:13672:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f0:0x0](ffff9023449a9988) for lock 0x4605c7c97e1c4a51 00000080:00010000:2.0:1788201214.388620:0:13672:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f0:0x0](ffff9023449a9988) 00000080:00200008:2.0:1788201214.388639:0:13672:0:(file.c:1090:ll_file_open()) START file file-7:[0x200000bd1:0x3f0:0x0](ffff9023449a9988), flags 104501 00000002:00100000:2.0:1788201214.388663:0:13672:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90235268b800 x1875064442618368/t8589938630(8589938630) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201230 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788201214.388706:0:13672:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4a51 from it ffff9023533eae40 00010000:00010000:2.0:1788201214.388710:0:13672:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3065 expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.388719:0:13672:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3065 expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201214.388747:0:13672:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f0:0x0](ffff9023449a9988), inode mode 81a4 mask 40 00000080:00200008:2.0:1788201214.388801:0:13672:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-7:[0x200000bd1:0x3f0:0x0](ffff9023449a9988) current size 0, valid attrs 20070, mode 9023, uid -2133541296, gid -24399, new size 18446744071973566999, atime 1788201214.000000000, mtime 1788201214.000000000, ctime 1788201214.000000000 00010000:00010000:2.0:1788201214.388823:0:13672:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d3065 expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201214.388844:0:13672:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3f0:0x0]: losing layout lock 00010000:00010000:2.0:1788201214.388929:0:13672:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3065 expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201214.388935:0:13672:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201214.388937:0:13672:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c4a51 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3065 expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788201214.388960:0:13672:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201214.388965:0:13672:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c5f80 x1875064442618624/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201214.388985:0:13672:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5f80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13672:1875064442618624:192.168.201.130@tcp:36:touch.0 00000100:00100000:2.0:1788201214.389055:0:13672:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448780 going to sleep for 11 seconds 00000100:00100000:2.0:1788201214.391351:0:13672:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5f80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13672:1875064442618624:192.168.201.130@tcp:36:touch.0 00000080:00200000:2.0:1788201214.391439:0:13672:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3f0:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:2.0:1788201214.391466:0:13672:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x3f0:0x0](ffff9023449a9988) 00000002:00010000:2.0:1788201214.391479:0:13672:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3f0:0x0]) in obj [0x200000bd1:0x3f0:0x0], intent: layout flags 0 00010000:00010000:2.0:1788201214.391539:0:13672:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c4a58 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 13672 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.391550:0:13672:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c4a58 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 13672 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.391562:0:13672:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c4a58 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 13672 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201214.391584:0:13672:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4000 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13672:1875064442618752:192.168.201.130@tcp:101:touch.0 00000100:00100000:2.0:1788201214.391657:0:13672:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448780 going to sleep for 11 seconds 00000100:00100000:2.0:1788201214.394072:0:13672:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4000 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13672:1875064442618752:192.168.201.130@tcp:101:touch.0 00010000:00010000:2.0:1788201214.394105:0:13672:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c4a58 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d307a expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.394120:0:13672:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c4a58 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d307a expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201214.394134:0:13672:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9023458c4000 x1875064442618752/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201230 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201214.394144:0:13672:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c4a58 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d307a expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201214.394156:0:13672:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c4a58 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d307a expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.394166:0:13672:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1008 (0 0) 00000080:00200000:2.0:1788201214.394173:0:13672:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x3f0:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788201214.394194:0:13672:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f0:0x0](ffff9023449a9988) for lock 0x4605c7c97e1c4a58 00000080:00010000:2.0:1788201214.394201:0:13672:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4a58 from it ffffa0b180d4ba78 00010000:00010000:2.0:1788201214.394205:0:13672:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c4a58 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d307a expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201214.394215:0:13672:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c4a58 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d307a expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201214.394242:0:13672:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c4a58 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d307a expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201214.394255:0:13672:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c4a58 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d307a expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788201214.394264:0:13672:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x3f0:0x0](00000000b4f69107) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c4a58 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d307a expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201214.394343:0:13672:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3f0:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:2.0:1788201214.394370:0:13672:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c4a58 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d307a expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201214.394378:0:13672:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c4a58 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d307a expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1788201214.394402:0:13672:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201214.394537:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442618880:192.168.201.130@tcp:2:touch.0 00000100:00100000:2.0:1788201214.396693:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442618880:192.168.201.130@tcp:2:touch.0 00000080:00200000:3.0:1788201214.396820:0:13672:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f0:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201214.396852:0:13672:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-7:[0x200000bd1:0x3f0:0x0](ffff9023449a9988) current size 0, valid attrs 100201f0, mode 9023, uid -2133541296, gid -24399, new size 18446744071973566999, atime 1788201214.000000000, mtime 1788201214.000000000, ctime 1788201214.000000000, rc 0 00000080:00200008:3.0:1788201214.396913:0:13672:0:(file.c:479:ll_file_release()) START file file-7:[0x200000bd1:0x3f0:0x0](ffff9023449a9988), flags 104001 00000002:00100000:3.0:1788201214.396992:0:13672:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff90235268b800 x1875064442618368/t8589938630(8589938630) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201230 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201214.397007:0:13672:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201214.397013:0:13672:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90235268aa00 x1875064442619136/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201214.397038:0:13672:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13672:1875064442619136:192.168.201.130@tcp:35:touch.0 00000100:00100000:3.0:1788201214.397113:0:13672:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe30c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201214.399706:0:13672:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13672:1875064442619136:192.168.201.130@tcp:35:touch.0 00000002:00100000:2.0:1788201214.399869:0:13672:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff90235268b800 x1875064442618368/t8589938630(8589938630) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201230 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201215.189812:0:13684:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201215.189848:0:13684:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201215.189872:0:13684:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788201215.189976:0:13684:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1788201215.189985:0:13684:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201215.189998:0:13684:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201215.190002:0:13684:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c49e1 00000080:00010000:3.0:1788201215.190012:0:13684:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49e1 from it ffffa0b181bebb58 00010000:00010000:3.0:1788201215.190017:0:13684:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788201215.190026:0:13684:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1788201215.190048:0:13684:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201215.190057:0:13684:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102a.replay-single, flags=25 00000080:00200000:3.0:1788201215.190063:0:13684:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 1 00000080:00200000:3.0:1788201215.190074:0:13684:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-2, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), flags=9 00000080:00200000:3.0:1788201215.190078:0:13684:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-2, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), intent=getattr 00000002:00010000:3.0:1788201215.190092:0:13684:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-2,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3e9:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201215.190149:0:13684:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 13684 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.190158:0:13684:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13684 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.190175:0:13684:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13684 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201215.190273:0:13684:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13684:1875064442619264:192.168.201.130@tcp:101:chmod.0 00000100:00100000:3.0:1788201215.190423:0:13684:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3d80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201215.195839:0:13684:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13684:1875064442619264:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201215.196022:0:13684:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.196046:0:13684:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.196056:0:13684:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.196072:0:13684:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.196088:0:13684:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.196104:0:13684:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.196113:0:13684:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.196125:0:13684:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201215.196139:0:13684:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff90235268b100 x1875064442619264/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201231 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201215.196154:0:13684:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201215.196169:0:13684:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.196181:0:13684:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1003 (0 0) 00000080:00200000:3.0:1788201215.196249:0:13684:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3e9:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201215.196370:0:13684:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3eb:0x0](ffff9023449b3248): rc = 0 00000080:00200000:3.0:1788201215.196393:0:13684:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3eb:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201215.196494:0:13684:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3eb:0x0](ffff9023449b3248) for lock 0x4605c7c97e1c4a5f 00000080:00010000:3.0:1788201215.196513:0:13684:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3eb:0x0](ffff9023449b3248) 00000080:00010000:3.0:1788201215.196517:0:13684:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4a5f from it ffffa0b181bebb20 00010000:00010000:3.0:1788201215.196521:0:13684:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.196532:0:13684:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788201215.196775:0:13684:0:(file.c:6283:ll_getattr_dentry()) START file file-2:[0x200000bd1:0x3eb:0x0](ffff9023449b3248), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201215.196785:0:13684:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3eb:0x0](ffff9023449b3248),name=file-2 00000002:00010000:3.0:1788201215.196810:0:13684:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3eb:0x0]) in obj [0x200000bd1:0x3e9:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201215.196821:0:13684:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201215.196834:0:13684:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201215.196841:0:13684:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3e9:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201215.196846:0:13684:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3eb:0x0](ffff9023449b3248) 00000080:00010000:3.0:1788201215.196850:0:13684:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3eb:0x0](ffff9023449b3248) for lock 0x4605c7c97e1c4a5f 00000080:00010000:3.0:1788201215.196854:0:13684:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4a5f from it ffffa0b181bebcb8 00010000:00010000:3.0:1788201215.196857:0:13684:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201215.196864:0:13684:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201215.196995:0:13684:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3eb:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1788201215.197008:0:13684:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3eb:0x0] 00000008:00010000:3.0:1788201215.197049:0:13684:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788201215.197054:0:13684:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023483ef630, osc lock ffff9023488677c0, flags 0x400001002 00010000:00010000:3.0:1788201215.197116:0:13684:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d90d9cc3/0x4605c7c97e1c4a66 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 13684 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.197127:0:13684:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d90d9cc3/0x4605c7c97e1c4a66 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 13684 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788201215.197170:0:13684:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1788201215.198120:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442619392:192.168.201.130@tcp:101:chmod.0 00000080:00200000:2.0:1788201215.199868:0:13686:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201215.205532:0:13686:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201215.205549:0:13686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788201215.205569:0:13686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1788201215.205577:0:13686:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201215.205584:0:13686:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201215.205588:0:13686:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c49e1 00000080:00010000:2.0:1788201215.205594:0:13686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49e1 from it ffffa0b181c23b58 00010000:00010000:2.0:1788201215.205598:0:13686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1788201215.205606:0:13686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:2.0:1788201215.205622:0:13686:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201215.205631:0:13686:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102a.replay-single, flags=25 00000080:00200000:2.0:1788201215.205638:0:13686:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201215.205648:0:13686:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-4, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), flags=9 00000080:00200000:2.0:1788201215.205652:0:13686:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-4, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), intent=getattr 00000002:00010000:2.0:1788201215.205662:0:13686:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-4,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3e9:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201215.205699:0:13686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 13686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.205707:0:13686:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 13686 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.205727:0:13686:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 13686 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201215.205757:0:13686:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13686:1875064442619648:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201215.206837:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d90d9cc3/0x4605c7c97e1c4a66 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 13684 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.206865:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d90d9cc3/0x4605c7c97e1c4a66 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d309d expref: -99 pid: 13684 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.206978:0:2362:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d90d9cc3/0x4605c7c97e1c4a66 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d309d expref: -99 pid: 13684 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.206994:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d90d9cc3/0x4605c7c97e1c4a66 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d309d expref: -99 pid: 13684 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1788201215.207016:0:2362:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d90d9cc3/0x4605c7c97e1c4a66 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d309d expref: -99 pid: 13684 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788201215.207030:0:2362:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1788201215.207045:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d90d9cc3/0x4605c7c97e1c4a66 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d309d expref: -99 pid: 13684 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.207055:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d90d9cc3/0x4605c7c97e1c4a66 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d309d expref: -99 pid: 13684 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.207064:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d90d9cc3/0x4605c7c97e1c4a66 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d309d expref: -99 pid: 13684 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.207079:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d90d9cc3/0x4605c7c97e1c4a66 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d309d expref: -99 pid: 13684 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1788201215.207100:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442619392:192.168.201.130@tcp:101:chmod.0 00000080:00200000:3.0:1788201215.207158:0:13684:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x3eb:0x0] updating i_size 0 i_blocks 1 00000020:00010000:3.0:1788201215.207164:0:13684:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235000ebf0 (ffff902346343878/0) at cl_lock_release():245 00000020:00010000:3.0:1788201215.207168:0:13684:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235000ebf0 (ffff902346343878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201215.207175:0:13684:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023483ef630 (ffff902346343c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788201215.207180:0:13684:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235000ebf0 (ffff902346343878/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201215.207183:0:13684:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023483ef630 (ffff902346343878/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201215.207294:0:13684:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3eb:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201215.207381:0:13684:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201215.207396:0:13684:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201215.207472:0:13684:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1788201215.207485:0:13684:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:3.0:1788201215.207492:0:13684:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201215.207500:0:13684:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201215.207503:0:13684:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c49e1 00000080:00010000:3.0:1788201215.207509:0:13684:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49e1 from it ffffa0b181bebc08 00010000:00010000:3.0:1788201215.207512:0:13684:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1788201215.207521:0:13684:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:3.0:1788201215.207531:0:13684:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201215.207538:0:13684:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102a.replay-single, flags=25 00000080:00200000:3.0:1788201215.207543:0:13684:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 1 00000080:00200000:3.0:1788201215.207547:0:13684:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-2, flags=9 00000080:00200008:3.0:1788201215.207562:0:13684:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-2:[0x200000bd1:0x3eb:0x0](ffff9023449b3248) current size 0, valid attrs 41, mode 8180, uid 1094316032, gid -28637, new size 0, atime 1788201215.000000000, mtime 1788201215.000000000, ctime 1788201215.000000000 00000100:00100000:2.0:1788201215.207658:0:13686:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448480 going to sleep for 11 seconds 00010000:00010000:3.0:1788201215.210551:0:13684:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201215.212968:0:13684:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c4a12 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d2f85 expref: -99 pid: 13666 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201215.213064:0:13684:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201215.213074:0:13684:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201215.213077:0:13684:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4a5f lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3096 expref: -99 pid: 13684 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201215.213098:0:13684:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201215.213116:0:13684:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90235268b100 x1875064442620032/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201215.213147:0:13684:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13684:1875064442620032:192.168.201.130@tcp:36:chmod.0 00000100:00100000:3.0:1788201215.215183:0:13684:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201215.226637:0:13686:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13686:1875064442619648:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201215.226652:0:13686:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.226663:0:13686:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.226670:0:13686:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.226677:0:13686:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.226686:0:13686:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.226694:0:13686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.226701:0:13686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.226707:0:13686:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201215.226721:0:13686:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c5880 x1875064442619648/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201231 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201215.226730:0:13686:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201215.226741:0:13686:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.226748:0:13686:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1005 (0 0) 00000080:00200000:2.0:1788201215.226753:0:13686:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3e9:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201215.226800:0:13686:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3ed:0x0](ffff9023449b00c8): rc = 0 00000080:00200000:2.0:1788201215.226815:0:13686:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3ed:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201215.226823:0:13686:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3ed:0x0](ffff9023449b00c8) for lock 0x4605c7c97e1c4a74 00000080:00010000:2.0:1788201215.226829:0:13686:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3ed:0x0](ffff9023449b00c8) 00000080:00010000:2.0:1788201215.226831:0:13686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4a74 from it ffffa0b181c23b20 00010000:00010000:2.0:1788201215.226834:0:13686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.226842:0:13686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788201215.226861:0:13686:0:(file.c:6283:ll_getattr_dentry()) START file file-4:[0x200000bd1:0x3ed:0x0](ffff9023449b00c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201215.232549:0:13689:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000080:00200000:2.0:1788201215.236532:0:13686:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3ed:0x0](ffff9023449b00c8),name=file-4 00000002:00010000:2.0:1788201215.236554:0:13686:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3ed:0x0]) in obj [0x200000bd1:0x3e9:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201215.236566:0:13686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201215.236578:0:13686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201215.236584:0:13686:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3e9:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201215.237895:0:13686:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3ed:0x0](ffff9023449b00c8) 00000080:00010000:2.0:1788201215.237902:0:13686:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3ed:0x0](ffff9023449b00c8) for lock 0x4605c7c97e1c4a74 00000080:00010000:2.0:1788201215.237909:0:13686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4a74 from it ffffa0b181c23cb8 00010000:00010000:2.0:1788201215.237913:0:13686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201215.237921:0:13686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201215.237968:0:13686:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3ed:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201215.237978:0:13686:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3ed:0x0] 00000008:00010000:2.0:1788201215.238006:0:13686:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201215.238009:0:13686:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398a30, osc lock ffff9023500dcbe0, flags 0x400001002 00010000:00010000:2.0:1788201215.238049:0:13686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.238059:0:13686:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201215.238122:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442621312:192.168.201.130@tcp:101:chmod.0 00000020:00200000:2.0:1788201215.238247:0:13686:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000002:00010000:2.0:1788201215.238270:0:13689:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201215.238281:0:13689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:2.0:1788201215.238292:0:13689:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:2.0:1788201215.238310:0:13689:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201215.238317:0:13689:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201215.238319:0:13689:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c49e1 00000080:00010000:2.0:1788201215.238323:0:13689:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49e1 from it ffffa0b181c3bb58 00010000:00010000:2.0:1788201215.238327:0:13689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:2.0:1788201215.238333:0:13689:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:2.0:1788201215.238344:0:13689:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201215.238353:0:13689:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102a.replay-single, flags=25 00000080:00200000:2.0:1788201215.238359:0:13689:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201215.238372:0:13689:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-7, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), flags=9 00000080:00200000:2.0:1788201215.238377:0:13689:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-7, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), intent=getattr 00010000:00010000:2.0:1788201215.238418:0:13691:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4add lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 9 type: IBT flags: 0x10000000000000 pid: 13691 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.238425:0:13691:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4add lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x0 pid: 13691 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201215.238512:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201215.238519:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442621696:192.168.201.130@tcp:63:ll_sa.0 00000002:00010000:2.0:1788201215.238578:0:13689:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-7,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3e9:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201215.238607:0:13689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 10 type: IBT flags: 0x10000000000000 pid: 13689 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.238621:0:13689:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 10 type: IBT flags: 0x0 pid: 13689 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.238630:0:13689:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 10 type: IBT flags: 0x0 pid: 13689 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201215.238643:0:13689:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13689:1875064442621824:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201215.239558:0:13693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4aeb lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 11 type: IBT flags: 0x10000000000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.239569:0:13693:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4aeb lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x0 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.239590:0:13693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4af2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 12 type: IBT flags: 0x10000000000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.239596:0:13693:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4af2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x0 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.239613:0:13693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201215.239622:0:13693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:3.0:1788201215.239629:0:13693:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4a74 from it ffffa0b181c63d78 00010000:00010000:3.0:1788201215.239633:0:13693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201215.239640:0:13693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201215.239678:0:13693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4af9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 13 type: IBT flags: 0x10000000000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.239685:0:13693:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4af9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x0 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.239699:0:13693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c4a90 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3130 expref: -99 pid: 13688 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201215.239706:0:13693:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c4a90 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3130 expref: -99 pid: 13688 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00010000:3.0:1788201215.240123:0:13693:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4a90 from it ffffa0b181c63d78 00010000:00010000:3.0:1788201215.240128:0:13693:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c4a90 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3130 expref: -99 pid: 13688 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201215.240136:0:13693:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c4a90 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3130 expref: -99 pid: 13688 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201215.240235:0:13693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 14 type: IBT flags: 0x10000000000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.240242:0:13693:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x0 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.240261:0:13693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4b07 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 15 type: IBT flags: 0x10000000000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.240268:0:13693:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4b07 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT flags: 0x0 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.240289:0:13693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4b0e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 16 type: IBT flags: 0x10000000000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.240296:0:13693:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4b0e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x0 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201215.240408:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442622208:192.168.201.130@tcp:63:ll_sa.0 00000100:00100000:2.0:1788201215.240829:0:13689:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448900 going to sleep for 11 seconds 00010000:00010000:3.0:1788201215.247786:0:2361:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.247825:0:2361:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.247844:0:2361:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.247857:0:2361:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1788201215.247944:0:2361:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788201215.247958:0:2361:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000100:00100000:2.0:1788201215.247960:0:13689:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13689:1875064442621824:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201215.247975:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.247983:0:13689:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 16 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.247991:0:2361:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.247997:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.248001:0:13689:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.248008:0:2361:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201215.248009:0:13689:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.248019:0:13689:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.248027:0:13689:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201215.248029:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442621312:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201215.248038:0:13689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.248049:0:13689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.248055:0:13689:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201215.248067:0:13689:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c5500 x1875064442621824/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201231 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201215.248080:0:13689:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788201215.248115:0:13686:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x3ed:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201215.248122:0:13686:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc4de0 (ffff902350b9d3f8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201215.248126:0:13686:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc4de0 (ffff902350b9d3f8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201215.248132:0:13686:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398a30 (ffff902350b9db48/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201215.248138:0:13686:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc4de0 (ffff902350b9d3f8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201215.248141:0:13686:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398a30 (ffff902350b9d3f8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201215.248184:0:13686:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3ed:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201215.248268:0:13686:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201215.248291:0:13686:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201215.248316:0:13686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:2.0:1788201215.248328:0:13686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:2.0:1788201215.248342:0:13686:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201215.248351:0:13686:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201215.248356:0:13686:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c49e1 00000080:00010000:2.0:1788201215.248362:0:13686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49e1 from it ffffa0b181c23c08 00010000:00010000:2.0:1788201215.248366:0:13686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:2.0:1788201215.248373:0:13686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:2.0:1788201215.248383:0:13686:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201215.248391:0:13686:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102a.replay-single, flags=25 00000080:00200000:2.0:1788201215.248397:0:13686:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201215.248401:0:13686:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-4, flags=9 00000080:00200008:2.0:1788201215.248441:0:13686:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-4:[0x200000bd1:0x3ed:0x0](ffff9023449b00c8) current size 0, valid attrs 41, mode 8180, uid 1204953088, gid -28637, new size 0, atime 1788201215.000000000, mtime 1788201215.000000000, ctime 1788201215.000000000 00010000:00010000:2.0:1788201215.248465:0:13686:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201215.248482:0:13686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c4a2e lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d2fe7 expref: -99 pid: 13668 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201215.248525:0:13686:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201215.248531:0:13686:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201215.248535:0:13686:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4a74 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d30d5 expref: -99 pid: 13686 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1788201215.248557:0:13686:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 5 is allocated opc 36, max 7 00000100:00100000:2.0:1788201215.248565:0:13686:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c7480 x1875064442622592/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201215.248591:0:13686:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7480 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13686:1875064442622592:192.168.201.130@tcp:36:chmod.0 00010000:00010000:3.0:1788201215.248733:0:13693:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4b15 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 16 type: IBT flags: 0x10000000000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.248741:0:13693:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4b15 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x0 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201215.248800:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201215.248814:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442622848:192.168.201.130@tcp:63:ll_sa.0 00000100:00100000:2.0:1788201215.249031:0:13686:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448900 going to sleep for 11 seconds 00000002:00010000:2.0:1788201215.249057:0:13689:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.249069:0:13689:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1008 (0 0) 00000080:00200000:2.0:1788201215.249074:0:13689:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3e9:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201215.249149:0:13689:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3f0:0x0](ffff9023449a9988): rc = 0 00000080:00200000:2.0:1788201215.249173:0:13689:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3f0:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201215.249184:0:13689:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f0:0x0](ffff9023449a9988) for lock 0x4605c7c97e1c4ae4 00000080:00010000:2.0:1788201215.249195:0:13689:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f0:0x0](ffff9023449a9988) 00000080:00010000:2.0:1788201215.249198:0:13689:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4ae4 from it ffffa0b181c3bb20 00010000:00010000:2.0:1788201215.249202:0:13689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.249210:0:13689:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788201215.249235:0:13689:0:(file.c:6283:ll_getattr_dentry()) START file file-7:[0x200000bd1:0x3f0:0x0](ffff9023449a9988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201215.249242:0:13689:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3f0:0x0](ffff9023449a9988),name=file-7 00000002:00010000:2.0:1788201215.249252:0:13689:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3f0:0x0]) in obj [0x200000bd1:0x3e9:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201215.249259:0:13689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201215.249269:0:13689:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201215.249275:0:13689:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3e9:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201215.249279:0:13689:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f0:0x0](ffff9023449a9988) 00000080:00010000:2.0:1788201215.249282:0:13689:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f0:0x0](ffff9023449a9988) for lock 0x4605c7c97e1c4ae4 00000080:00010000:2.0:1788201215.249285:0:13689:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4ae4 from it ffffa0b181c3bcb8 00010000:00010000:2.0:1788201215.249289:0:13689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201215.249296:0:13689:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201215.249338:0:13689:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3f0:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201215.249348:0:13689:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3f0:0x0] 00000008:00010000:2.0:1788201215.249377:0:13689:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201215.249382:0:13689:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398a30, osc lock ffff9023500dcbe0, flags 0x400001002 00010000:00010000:2.0:1788201215.249422:0:13689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000035fa0f84/0x4605c7c97e1c4b1c lrc: 3/1,0 mode: --/PR res: [0x240000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 13689 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.249433:0:13689:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000035fa0f84/0x4605c7c97e1c4b1c lrc: 3/1,0 mode: --/PR res: [0x240000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 13689 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201215.250045:0:13689:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1788201215.250054:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201215.250067:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442622976:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201215.253135:0:2361:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4aeb lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31b5 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.253147:0:2361:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4aeb lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31b5 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.253154:0:2361:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4aeb lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31b5 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.253163:0:2361:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4aeb lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31b5 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.253172:0:2361:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4aeb lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31b5 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.253181:0:2361:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4aeb lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31b5 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.253188:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4aeb lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31b5 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.253195:0:2361:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4aeb lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31b5 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201215.253205:0:2361:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff90235268a300 x1875064442622208/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 2624/6040 e 0 to 0 dl 1788201231 ref 1 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:2.0:1788201215.253214:0:2361:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4aeb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31b5 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:2.0:1788201215.253227:0:2361:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4aeb from it ffff902342dfa990 00010000:00010000:2.0:1788201215.253231:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4aeb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31b5 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.253240:0:2361:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4aeb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31b5 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201215.253270:0:2361:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4af2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31d1 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.253280:0:2361:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4af2 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3ec:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31d1 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.253287:0:2361:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4af2 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3ec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31d1 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.253293:0:2361:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4af2 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31d1 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254121:0:2361:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4af2 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31d1 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254133:0:2361:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4af2 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3ec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31d1 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254140:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4af2 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3ec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31d1 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254152:0:2361:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4af2 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31d1 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201215.254161:0:2361:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff90235268a300 x1875064442622208/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 2624/6040 e 0 to 0 dl 1788201231 ref 2 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:2.0:1788201215.254170:0:2361:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4af2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31d1 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:2.0:1788201215.254180:0:2361:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4af2 from it ffff902342df9d90 00010000:00010000:2.0:1788201215.254183:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4af2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31d1 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254190:0:2361:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4af2 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31d1 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201215.254205:0:2361:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4af9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31ed expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254215:0:2361:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4af9 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31ed expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254221:0:2361:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4af9 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31ed expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254228:0:2361:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4af9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31ed expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254240:0:2361:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4af9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31ed expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254247:0:2361:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4af9 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31ed expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254254:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4af9 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31ed expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254261:0:2361:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4af9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31ed expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201215.254270:0:2361:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff90235268a300 x1875064442622208/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 2624/6040 e 0 to 0 dl 1788201231 ref 3 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:2.0:1788201215.254278:0:2361:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4af9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31ed expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:2.0:1788201215.254286:0:2361:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4af9 from it ffff902342df8590 00010000:00010000:2.0:1788201215.254290:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4af9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31ed expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254296:0:2361:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4af9 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d31ed expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201215.254323:0:2361:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3209 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254336:0:2361:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3209 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254343:0:2361:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3209 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254350:0:2361:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3209 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254358:0:2361:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3209 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254365:0:2361:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3209 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254373:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3209 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254380:0:2361:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3209 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201215.254389:0:2361:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff90235268a300 x1875064442622208/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 2624/6040 e 0 to 0 dl 1788201231 ref 4 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:2.0:1788201215.254398:0:2361:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3209 expref: -99 pid: 13693 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:2.0:1788201215.254537:0:2361:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4b00 from it ffff902342dfb990 00010000:00010000:2.0:1788201215.254540:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3209 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254548:0:2361:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3209 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201215.254563:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4b07 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254570:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4b07 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254575:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4b07 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254581:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4b07 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254587:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4b07 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254593:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db0c00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201215.254597:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4b07 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254603:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4b07 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254613:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4b07 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x20e400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254619:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201215.254622:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4b07 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254629:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4b07 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254645:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4b0e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254651:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4b0e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254656:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4b0e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254662:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4b0e lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254667:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4b0e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254672:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db0a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201215.254675:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4b0e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254679:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4b0e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254686:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4b0e lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x20e400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254691:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201215.254694:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4b0e lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254701:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4b0e lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201215.254718:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442622208:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:2.0:1788201215.254753:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4add lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 13691 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254759:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4add lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 13691 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254764:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4add lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 13691 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254770:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4add lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 13691 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254776:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4add lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 13691 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254781:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351685e00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201215.254783:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4add lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 13691 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254793:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4add lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 13691 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254798:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4add lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x20e400000000 pid: 13691 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254802:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201215.254805:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4add lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 13691 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.254811:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4add lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 13691 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201215.254827:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442621696:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:2.0:1788201215.254869:0:11305:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4aeb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31b5 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201215.254974:0:11305:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3eb:0x0](ffff9023449b3248): rc = 0 00000080:00200000:2.0:1788201215.254990:0:11305:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3eb:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201215.254999:0:11305:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3eb:0x0](ffff9023449b3248) for lock 0x4605c7c97e1c4aeb 00000080:00010000:2.0:1788201215.255004:0:11305:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4aeb from it ffff902342dfa990 00010000:00010000:2.0:1788201215.255007:0:11305:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4aeb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d31b5 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201215.255016:0:11305:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4aeb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d31b5 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201215.255032:0:11305:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4af2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31d1 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201215.255063:0:11305:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3ec:0x0](ffff9023449a3248): rc = 0 00000080:00200000:2.0:1788201215.255076:0:11305:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3ec:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201215.255084:0:11305:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3ec:0x0](ffff9023449a3248) for lock 0x4605c7c97e1c4af2 00000080:00010000:2.0:1788201215.255088:0:11305:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4af2 from it ffff902342df9d90 00010000:00010000:2.0:1788201215.255091:0:11305:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4af2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d31d1 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201215.255098:0:11305:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4af2 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d31d1 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201215.255113:0:11305:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4af9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d31ed expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201215.255142:0:11305:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3ee:0x0](ffff9023449aa1c8): rc = 0 00000080:00200000:2.0:1788201215.255155:0:11305:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3ee:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201215.255164:0:11305:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3ee:0x0](ffff9023449aa1c8) for lock 0x4605c7c97e1c4af9 00000080:00010000:2.0:1788201215.255167:0:11305:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4af9 from it ffff902342df8590 00010000:00010000:2.0:1788201215.255170:0:11305:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4af9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d31ed expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201215.255178:0:11305:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4af9 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d31ed expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201215.255192:0:11305:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3209 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201215.255226:0:11305:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3f0:0x0](ffff9023449a9988): rc = 0 00000080:00200000:2.0:1788201215.255239:0:11305:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3f0:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201215.255249:0:11305:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f0:0x0](ffff9023449a9988) for lock 0x4605c7c97e1c4b00 00000080:00010000:2.0:1788201215.255252:0:11305:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4b00 from it ffff902342dfb990 00010000:00010000:2.0:1788201215.255255:0:11305:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3209 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201215.255263:0:11305:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3209 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201215.255729:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000035fa0f84/0x4605c7c97e1c4b1c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 13689 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.255750:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000035fa0f84/0x4605c7c97e1c4b1c lrc: 6/2,0 mode: --/PR res: [0x240000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d323a expref: -99 pid: 13689 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.255766:0:2362:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000035fa0f84/0x4605c7c97e1c4b1c lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d323a expref: -99 pid: 13689 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.255779:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000035fa0f84/0x4605c7c97e1c4b1c lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d323a expref: -99 pid: 13689 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201215.255795:0:2362:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000035fa0f84/0x4605c7c97e1c4b1c lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d323a expref: -99 pid: 13689 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201215.255815:0:2362:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788201215.255836:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000035fa0f84/0x4605c7c97e1c4b1c lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d323a expref: -99 pid: 13689 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.255844:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000035fa0f84/0x4605c7c97e1c4b1c lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d323a expref: -99 pid: 13689 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.255851:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000035fa0f84/0x4605c7c97e1c4b1c lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d323a expref: -99 pid: 13689 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201215.255859:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000035fa0f84/0x4605c7c97e1c4b1c lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d323a expref: -99 pid: 13689 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201215.255874:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442622976:192.168.201.130@tcp:101:chmod.0 00000080:00200000:3.0:1788201215.255987:0:13689:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x3f0:0x0] updating i_size 0 i_blocks 1 00000020:00010000:3.0:1788201215.255994:0:13689:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc4de0 (ffff902350b9d3f8/0) at cl_lock_release():245 00000020:00010000:3.0:1788201215.255998:0:13689:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc4de0 (ffff902350b9d3f8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201215.256003:0:13689:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398a30 (ffff902350b9db48/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788201215.256008:0:13689:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc4de0 (ffff902350b9d3f8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201215.256011:0:13689:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398a30 (ffff902350b9d3f8/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201215.256040:0:13689:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f0:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201215.256121:0:13689:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201215.256142:0:13689:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201215.256159:0:13689:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:3.0:1788201215.256172:0:13689:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:3.0:1788201215.256182:0:13689:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201215.256190:0:13689:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201215.256194:0:13689:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c49e1 00000080:00010000:3.0:1788201215.256199:0:13689:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49e1 from it ffffa0b181c3bc08 00010000:00010000:3.0:1788201215.256203:0:13689:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:3.0:1788201215.256211:0:13689:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:3.0:1788201215.256222:0:13689:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201215.256230:0:13689:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102a.replay-single, flags=25 00000080:00200000:3.0:1788201215.256236:0:13689:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 1 00000080:00200000:3.0:1788201215.256240:0:13689:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-7, flags=9 00000080:00200008:3.0:1788201215.256259:0:13689:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-7:[0x200000bd1:0x3f0:0x0](ffff9023449a9988) current size 0, valid attrs 41, mode 8180, uid 1094316032, gid -28637, new size 0, atime 1788201215.000000000, mtime 1788201215.000000000, ctime 1788201215.000000000 00010000:00010000:3.0:1788201215.256282:0:13689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d3209 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201215.256310:0:13689:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c4a58 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x8/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d307a expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201215.256330:0:13689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c4aac lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d3184 expref: -99 pid: 13691 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201215.256341:0:13689:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c4a58 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x8/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d307a expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201215.256352:0:13689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201215.256363:0:13689:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c4a58 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x8/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d307a expref: -99 pid: 13672 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788201215.256400:0:13689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3209 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201215.256409:0:13689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c4aac lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3184 expref: -99 pid: 13691 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201215.256416:0:13689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201215.256421:0:13689:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:3.0:1788201215.256425:0:13689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b00 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3209 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201215.256437:0:13689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c4aac lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3184 expref: -99 pid: 13691 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201215.256452:0:13689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4ae4 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3168 expref: -99 pid: 13689 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201215.256469:0:13689:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 7 is allocated opc 36, max 7 00000100:00100000:3.0:1788201215.256479:0:13689:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90235268b480 x1875064442623232/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 496/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201215.256506:0:13689:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13689:1875064442623232:192.168.201.130@tcp:36:chmod.0 00000100:00100000:3.0:1788201215.256604:0:13689:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3300 going to sleep for 11 seconds 00010000:00010000:3.0:1788201215.266210:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4b15 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.266230:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4b15 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.266234:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4b15 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.266239:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4b15 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.266246:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4b15 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.266250:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db0400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201215.266254:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4b15 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.266259:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4b15 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.266265:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4b15 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x20e400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.266270:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201215.266277:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4b15 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201215.266280:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4b15 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 13693 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201215.266321:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442622848:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:2.0:1788201215.361517:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201218.433704:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442623360:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201218.433843:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442623616:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201218.433906:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201218.433912:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442623488:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201218.433933:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442623744:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201218.444264:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442623488:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201218.444287:0:2362:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff902350a11000: committing for last_committed 4294968836 gen 2 00000100:00100000:3.0:1788201218.444314:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442623744:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201218.444324:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442623360:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201218.444345:0:2362:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0000-osc-ffff902350a11000: committing for last_committed 4294968835 gen 2 00000100:00100000:3.0:1788201218.444430:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442623616:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201219.462028:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201223.553299:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442624000:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201223.553451:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442624256:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201223.553495:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788201223.553501:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442623872:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201223.553531:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442624128:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201223.576026:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442623872:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201223.576057:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442624128:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201223.576069:0:2361:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 8589938632 gen 2 00000100:00100000:3.0:1788201223.580987:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442624000:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201223.581057:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442624256:192.168.201.130@tcp:400:kworker.0 00010000:00010000:3.0:1788201224.579214:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201226.625039:0:13689:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3300 going to sleep for 1 seconds 00000100:00100000:2.0:1788201226.625076:0:13684:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3d80 going to sleep for 1 seconds 00000100:00100000:2.0:1788201226.625115:0:13685:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2cc0 going to sleep for 1 seconds 00010000:00010000:3.0:1788201226.625148:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201227.649074:0:13685:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2cc0 going to sleep for 1 seconds 00000100:00100000:2.0:1788201227.649227:0:13684:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3d80 going to sleep for 1 seconds 00000100:00100000:2.0:1788201227.649247:0:13689:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3300 going to sleep for 1 seconds 00010000:00010000:3.0:1788201227.649343:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201227.649382:0:13686:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448900 going to sleep for 1 seconds 00000100:00100000:3.0:1788201227.649404:0:13683:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bcc0 going to sleep for 1 seconds 00000100:00100000:2.0:1788201227.649418:0:13687:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2900 going to sleep for 1 seconds 00000100:00100000:2.0:1788201228.673179:0:13684:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3d80 going to sleep for 1 seconds 00000100:00100000:3.0:1788201228.673199:0:13683:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bcc0 going to sleep for 1 seconds 00000100:00100000:3.0:1788201228.673218:0:13686:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448900 going to sleep for 1 seconds 00010000:00010000:3.0:1788201228.673261:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201228.673272:0:13687:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2900 going to sleep for 1 seconds 00000100:00100000:3.0:1788201229.697112:0:13687:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2900 going to sleep for 1 seconds 00010000:00010000:3.0:1788201229.698677:0:13082:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b2a lrc: 3/1,0 mode: --/CR res: [0x200000401:0x4:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 13082 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201229.698693:0:13082:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b2a lrc: 3/1,0 mode: --/CR res: [0x200000401:0x4:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x0 pid: 13082 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201229.698777:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442625024:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:2.0:1788201229.700695:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201229.702269:0:13685:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2cc0 going to sleep for 1 seconds 00010000:00010000:2.0:1788201229.713161:0:2362:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b2a lrc: 4/1,0 mode: --/CR res: [0x200000401:0x4:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x0 pid: 13082 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201229.713182:0:2362:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b2a lrc: 4/1,0 mode: --/CR res: [0x200000401:0x4:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 13082 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201229.713200:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b2a lrc: 4/1,0 mode: --/CR res: [0x200000401:0x4:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 13082 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201229.713207:0:2362:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b2a lrc: 3/0,0 mode: --/CR res: [0x200000401:0x4:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 13082 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201229.713216:0:2362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b2a lrc: 4/0,0 mode: --/CR res: [0x200000401:0x4:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 13082 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201229.713223:0:2362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db1400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201229.713228:0:2362:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b2a lrc: 4/0,0 mode: --/CR res: [0x200000401:0x4:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x206400000000 pid: 13082 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201229.713235:0:2362:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b2a lrc: 4/0,0 mode: --/CR res: [0x200000401:0x4:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x206400000000 pid: 13082 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201229.713244:0:2362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b2a lrc: 5/0,0 mode: --/CR res: [0x200000401:0x4:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x20e400000000 pid: 13082 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201229.713250:0:2362:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201229.713258:0:2362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b2a lrc: 3/0,0 mode: --/CR res: [0x200000401:0x4:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 13082 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201229.713264:0:2362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4b2a lrc: 0/0,0 mode: --/CR res: [0x200000401:0x4:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 13082 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201229.713307:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442625024:192.168.201.130@tcp:63:ll_sa.0 00000080:00200000:2.0:1788201229.833032:0:13082:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000401:0x1:0x0](ffff902352173248) 00000080:00200000:2.0:1788201229.833079:0:13082:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000401:0x4:0x0](ffff90235217db88) 00010000:00010000:3.0:1788201230.721215:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201230.722769:0:13688:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2540 going to sleep for 1 seconds 00000100:00100000:3.0:1788201230.722819:0:13689:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3300 going to sleep for 1 seconds 00000100:00100000:2.0:1788201230.722898:0:13684:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3d80 going to sleep for 1 seconds 00000100:00100000:3.0:1788201230.723013:0:13686:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448900 going to sleep for 1 seconds 00000100:00000400:2.0:1788201231.746345:0:13684:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201215/real 1788201215] req@ffff90235268b100 x1875064442620032/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201231 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1788201231.746383:0:13686:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201215/real 1788201215] req@ffff9023458c7480 x1875064442622592/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201231 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201231.789906:0:13684:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 2, was 2): DISCONN 00000100:00100000:2.0:1788201231.789913:0:13684:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3d80 going to sleep for 0 seconds 00000100:00080000:3.0:1788201231.792188:0:13686:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 2, was 2): DISCONN 00000100:00100000:3.0:1788201231.792203:0:13686:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448900 going to sleep for 0 seconds 00000100:00000400:3.0:1788201231.792252:0:13689:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201215/real 1788201215] req@ffff90235268b480 x1875064442623232/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 496/512 e 0 to 1 dl 1788201231 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201231.792282:0:13689:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 2, was 2): DISCONN 00000100:00100000:3.0:1788201231.792285:0:13689:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3300 going to sleep for 0 seconds 00000100:00000400:3.0:1788201231.792308:0:13688:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201215/real 1788201215] req@ffff902345831c00 x1875064442622720/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201231 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201231.792322:0:13688:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 2, was 2): DISCONN 00000100:00100000:3.0:1788201231.792325:0:13688:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2540 going to sleep for 0 seconds 00010000:00010000:3.0:1788201231.792374:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201231.792673:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201231.792681:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201231.792685:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000400:3.0:1788201231.793093:0:13685:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201215/real 1788201215] req@ffff902345830380 x1875064442620672/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201231 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201231.794515:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 392 00000100:00080000:2.0:1788201231.794535:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788201231.794549:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201231.794580:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023458c4a80 x1875064442625280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201231.794726:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442625408:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201231.795177:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788201231.795186:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442625152:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201231.795212:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442625536:192.168.201.130@tcp:400:kworker.0 00000100:00080000:3.0:1788201231.799283:0:13685:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 3, was 2): CONNECTING 00000100:00100000:2.0:1788201231.799404:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442625152:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201231.799427:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442625536:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201231.799439:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442625408:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201231.805530:0:13685:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2cc0 going to sleep for 0 seconds 00000100:02000000:2.0:1788201231.816370:0:2358:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff902350a11000: Connection restored to 192.168.201.130@tcp (at 192.168.201.130@tcp) 00000100:00080000:2.0:1788201231.829627:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000025066097): req@ffff902345830000 x1875064442622080/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201231 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201231.829647:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000005123105a): req@ffff90235268b100 x1875064442620032/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201231 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201231.829668:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000009d83cf48): req@ffff9023458c7480 x1875064442622592/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201231 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201231.829680:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000b6a31f6f): req@ffff90235268b480 x1875064442623232/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 496/512 e 0 to 1 dl 1788201231 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201231.829689:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000cc9987b2): req@ffff902345831c00 x1875064442622720/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201231 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201231.829699:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000004048d7aa): req@ffff9023488b1180 x1875064442621568/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201231 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201231.829710:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000b2a959d8): req@ffff902345830380 x1875064442620672/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201231 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201231.829725:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442625280:192.168.201.130@tcp:38:kworker.0 00000100:00080000:2.0:1788201231.829826:0:13686:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442622592 new x1875064442625664 00000100:00080000:3.0:1788201231.829843:0:13688:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442622720 new x1875064442625792 00000100:00100000:2.0:1788201231.831254:0:13686:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448900 going to sleep for 11 seconds 00000100:00100000:3.0:1788201231.836237:0:13688:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2540 going to sleep for 11 seconds 00000100:00080000:3.0:1788201231.836295:0:13689:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442623232 new x1875064442626176 00000100:00100000:3.0:1788201231.836352:0:13689:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3300 going to sleep for 11 seconds 00000100:00080000:3.0:1788201231.838646:0:13684:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442620032 new x1875064442626432 00000100:00100000:3.0:1788201231.839100:0:13684:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3d80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201231.839638:0:13686:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7480 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13686:1875064442622592:192.168.201.130@tcp:36:chmod.0 00000080:00200000:3.0:1788201231.839748:0:13686:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3ed:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000100:00100000:3.0:1788201231.849429:0:13688:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345831c00 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13688:1875064442622720:192.168.201.130@tcp:36:chmod.0 00000080:00200000:3.0:1788201231.849570:0:13688:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3ef:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201231.849586:0:13688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201231.849690:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201231.849711:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442626688:192.168.201.130@tcp:2:chmod.0 00000100:00100000:2.0:1788201231.862472:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201231.862506:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458216832 00000100:00100000:2.0:1788201231.862515:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458216832 00000100:00100000:2.0:1788201231.862558:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1309 00000100:00100000:2.0:1788201231.862567:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c4a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:35604eb4-baf9-45be-a20a-4883a2cd8cf9+5:13379:x1875064458216832:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201231.862585:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4af9 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x9383f985bc4d31ed expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201231.862663:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201231.862669:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c4a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:35604eb4-baf9-45be-a20a-4883a2cd8cf9+5:13379:x1875064458216832:12345-192.168.201.130@tcp:104: Request processed in 103us (206us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201231.862676:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1309 00000080:00200000:2.0:1788201231.863258:0:13689:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f0:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201231.863378:0:13689:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-7:[0x200000bd1:0x3f0:0x0](ffff9023449a9988) current size 0, valid attrs 10000041, mode 8180, uid 1094316032, gid -28637, new size 0, atime 1788201215.000000000, mtime 1788201215.000000000, ctime 1788201215.000000000, rc 0 00000100:00100000:3.0:1788201231.875373:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442626688:192.168.201.130@tcp:2:chmod.0 00000080:00200000:3.0:1788201231.875440:0:13688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3ef:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201231.875463:0:13688:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-6:[0x200000bd1:0x3ef:0x0](ffff90234497db88) current size 0, valid attrs 10000041, mode 8180, uid 1436811264, gid -28637, new size 0, atime 1788201215.000000000, mtime 1788201215.000000000, ctime 1788201215.000000000, rc 0 00000100:00100000:3.0:1788201231.883340:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201231.883381:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458216960 00000100:00100000:3.0:1788201231.883389:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458216960 00000100:00100000:3.0:1788201231.883442:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1310 00000100:00100000:3.0:1788201231.883451:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90235268a680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:35604eb4-baf9-45be-a20a-4883a2cd8cf9+5:12971:x1875064458216960:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201231.883472:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4aeb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3eb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x9383f985bc4d31b5 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201231.883534:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201231.883540:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90235268a680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:35604eb4-baf9-45be-a20a-4883a2cd8cf9+5:12971:x1875064458216960:12345-192.168.201.130@tcp:104: Request processed in 90us (209us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201231.883547:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1310 00000100:00100000:3.0:1788201231.883600:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201231.890267:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458217088 00000100:00100000:2.0:1788201231.890338:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458217088 00000100:00100000:2.0:1788201231.890409:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1311 00000100:00100000:2.0:1788201231.890417:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902352689180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:35604eb4-baf9-45be-a20a-4883a2cd8cf9+5:12969:x1875064458217088:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201231.890437:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4af2 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x9383f985bc4d31d1 expref: -99 pid: 13693 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201231.890532:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201231.890538:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902352689180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:35604eb4-baf9-45be-a20a-4883a2cd8cf9+5:12969:x1875064458217088:12345-192.168.201.130@tcp:104: Request processed in 122us (6937us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201231.890545:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1311 00000080:00200000:2.0:1788201231.905150:0:13687:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3ee:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201231.905185:0:13687:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-5:[0x200000bd1:0x3ee:0x0](ffff9023449aa1c8) current size 0, valid attrs 10000041, mode 8180, uid 1436811264, gid -28637, new size 0, atime 1788201215.000000000, mtime 1788201215.000000000, ctime 1788201215.000000000, rc 0 00000100:00100000:3.0:1788201231.919662:0:13684:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13684:1875064442620032:192.168.201.130@tcp:36:chmod.0 00000080:00200000:3.0:1788201231.919758:0:13684:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3eb:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201231.919774:0:13684:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201231.919897:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442627584:192.168.201.130@tcp:2:chmod.0 00000100:00100000:2.0:1788201231.923514:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442627584:192.168.201.130@tcp:2:chmod.0 00000080:00200000:3.0:1788201231.923607:0:13684:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3eb:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201231.923632:0:13684:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-2:[0x200000bd1:0x3eb:0x0](ffff9023449b3248) current size 0, valid attrs 10000041, mode 8180, uid 1094316032, gid -28637, new size 0, atime 1788201215.000000000, mtime 1788201215.000000000, ctime 1788201215.000000000, rc 0 00000080:00200000:3.0:1788201231.928520:0:13685:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3ec:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201231.928543:0:13685:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-3:[0x200000bd1:0x3ec:0x0](ffff9023449a3248) current size 0, valid attrs 10000041, mode 8180, uid 1436835840, gid -28637, new size 0, atime 1788201215.000000000, mtime 1788201215.000000000, ctime 1788201215.000000000, rc 0 00000080:00200000:2.0:1788201232.036099:0:13714:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201232.036464:0:13714:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201232.036486:0:13714:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1788201232.036502:0:13714:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:2.0:1788201232.036510:0:13714:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201232.036519:0:13714:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201232.036523:0:13714:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c49e1 00000080:00010000:2.0:1788201232.036531:0:13714:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49e1 from it ffffa0b181cb3b58 00010000:00010000:2.0:1788201232.036536:0:13714:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1788201232.036546:0:13714:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:2.0:1788201232.036563:0:13714:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201232.036571:0:13714:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102a.replay-single, flags=24 00000080:00200000:2.0:1788201232.036577:0:13714:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201232.036591:0:13714:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-4, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), flags=8 00000080:00200000:2.0:1788201232.036595:0:13714:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-4, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), intent=getattr 00000002:00010000:2.0:1788201232.036628:0:13714:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-4,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3e9:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201232.036674:0:13714:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 11 type: IBT flags: 0x10000000000000 pid: 13714 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.036683:0:13714:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 11 type: IBT flags: 0x0 pid: 13714 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.036695:0:13714:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 11 type: IBT flags: 0x0 pid: 13714 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201232.036721:0:13714:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13714:1875064442628352:192.168.201.130@tcp:101:checkstat.0 00000100:00100000:2.0:1788201232.036830:0:13714:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448240 going to sleep for 11 seconds 00000100:00100000:2.0:1788201232.056858:0:13714:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13714:1875064442628352:192.168.201.130@tcp:101:checkstat.0 00010000:00010000:2.0:1788201232.057252:0:13714:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.057271:0:13714:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.057280:0:13714:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.057293:0:13714:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.057305:0:13714:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.057319:0:13714:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.057327:0:13714:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.057411:0:13714:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201232.057425:0:13714:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c5500 x1875064442628352/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201248 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201232.057437:0:13714:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201232.057451:0:13714:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.057464:0:13714:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c4b46 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d32fe expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201232.057476:0:13714:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c4b46 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d32fe expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201232.057485:0:13714:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.057491:0:13714:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.057498:0:13714:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.057575:0:13714:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1788201232.057585:0:13714:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3e9:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201232.057674:0:13714:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3ed:0x0](ffff9023449b00c8): rc = 0 00000080:00200000:2.0:1788201232.057696:0:13714:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3ed:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201232.057707:0:13714:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3ed:0x0](ffff9023449b00c8) for lock 0x4605c7c97e1c4b46 00000080:00010000:2.0:1788201232.057720:0:13714:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3ed:0x0](ffff9023449b00c8) 00000080:00010000:2.0:1788201232.057724:0:13714:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4b46 from it ffffa0b181cb3b20 00010000:00010000:2.0:1788201232.057729:0:13714:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c4b46 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d32fe expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201232.057736:0:13714:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c4b46 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d32fe expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1788201232.057772:0:13714:0:(file.c:6283:ll_getattr_dentry()) START file file-4:[0x200000bd1:0x3ed:0x0](ffff9023449b00c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201232.057780:0:13714:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3ed:0x0](ffff9023449b00c8),name=file-4 00000002:00010000:2.0:1788201232.057813:0:13714:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3ed:0x0]) in obj [0x200000bd1:0x3e9:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201232.057821:0:13714:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c4b46 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d32fe expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201232.057829:0:13714:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c4b46 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d32fe expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201232.057843:0:13714:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3e9:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201232.057848:0:13714:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3ed:0x0](ffff9023449b00c8) 00000080:00010000:2.0:1788201232.057851:0:13714:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3ed:0x0](ffff9023449b00c8) for lock 0x4605c7c97e1c4b46 00000080:00010000:2.0:1788201232.057855:0:13714:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4b46 from it ffffa0b181cb3cb8 00010000:00010000:2.0:1788201232.057859:0:13714:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c4b46 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d32fe expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201232.057866:0:13714:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c4b46 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d32fe expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201232.057923:0:13714:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3ed:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201232.057933:0:13714:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3ed:0x0] 00000008:00010000:2.0:1788201232.057961:0:13714:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201232.057964:0:13714:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398ab0, osc lock ffff9023500dcd10, flags 0x400001002 00010000:00010000:2.0:1788201232.057982:0:13714:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x417:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201232.057992:0:13714:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788201232.058004:0:13714:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788201232.058017:0:13714:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201232.058026:0:13714:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1788201232.058036:0:13714:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788201232.058048:0:13714:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x3ed:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201232.058052:0:13714:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_release():245 00000020:00010000:2.0:1788201232.058055:0:13714:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201232.058058:0:13714:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398ab0 (ffff902350b9d758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201232.058062:0:13714:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201232.058065:0:13714:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398ab0 (ffff902350b9d638/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201232.058084:0:13714:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3ed:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1788201232.063278:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201232.063293:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.063307:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351685400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201232.063313:0:13086:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.063329:0:13086:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.063442:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 3/0,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.067097:0:13086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c4b46 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d32fe expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201232.067252:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 2/0,0 mode: --/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.067261:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201232.067326:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442628736:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201232.067447:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.067457:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4b7e lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3ed:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3398 expref: -99 pid: 13714 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.067482:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201232.067485:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201232.067487:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201232.072691:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442628736:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201232.074870:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201232.075187:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c4b8c lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d33b4 expref: -99 pid: 13715 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.075202:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839de00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201232.075210:0:13084:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c4b8c lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d33b4 expref: -99 pid: 13715 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.075220:0:13084:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c4b8c lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d33b4 expref: -99 pid: 13715 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.075230:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c4b8c lrc: 3/0,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d33b4 expref: -99 pid: 13715 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.075259:0:13084:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c4b4d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d331a expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201232.075312:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c4b8c lrc: 2/0,0 mode: --/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d33b4 expref: -99 pid: 13715 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.075441:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201232.075548:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c4b8c lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d33b4 expref: -99 pid: 13715 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.075558:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c4b8c lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3ee:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d33b4 expref: -99 pid: 13715 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.075584:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201232.075586:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201232.075590:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201232.075991:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442628864:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201232.078123:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442628864:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201232.108828:0:13716:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4b54 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3336 expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201232.108860:0:13716:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c4b93 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33d0 expref: -99 pid: 13716 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.108870:0:13716:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c4b93 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d33d0 expref: -99 pid: 13716 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.108928:0:13716:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c4b93 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d33d0 expref: -99 pid: 13716 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.108959:0:13716:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1788201232.108973:0:13716:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3e9:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201232.109231:0:13716:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3ef:0x0](ffff90234497db88): rc = 0 00000080:00200000:2.0:1788201232.109255:0:13716:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3ef:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201232.109267:0:13716:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3ef:0x0](ffff90234497db88) for lock 0x4605c7c97e1c4b54 00000080:00010000:2.0:1788201232.109281:0:13716:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3ef:0x0](ffff90234497db88) 00000080:00010000:2.0:1788201232.109285:0:13716:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4b54 from it ffffa0b181c9bb20 00010000:00010000:2.0:1788201232.109289:0:13716:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4b54 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3336 expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201232.109299:0:13716:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4b54 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3336 expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1788201232.109571:0:13716:0:(file.c:6283:ll_getattr_dentry()) START file file-6:[0x200000bd1:0x3ef:0x0](ffff90234497db88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201232.109581:0:13716:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3ef:0x0](ffff90234497db88),name=file-6 00000002:00010000:2.0:1788201232.109605:0:13716:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3ef:0x0]) in obj [0x200000bd1:0x3e9:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201232.109621:0:13716:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4b54 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3336 expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201232.109632:0:13716:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4b54 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3336 expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201232.109640:0:13716:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3e9:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201232.109645:0:13716:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3ef:0x0](ffff90234497db88) 00000080:00010000:2.0:1788201232.109649:0:13716:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3ef:0x0](ffff90234497db88) for lock 0x4605c7c97e1c4b54 00000080:00010000:2.0:1788201232.109653:0:13716:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4b54 from it ffffa0b181c9bcb8 00010000:00010000:2.0:1788201232.109656:0:13716:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4b54 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3336 expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201232.109664:0:13716:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4b54 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ef:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3336 expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201232.110436:0:13716:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3ef:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201232.110451:0:13716:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3ef:0x0] 00000008:00010000:2.0:1788201232.110852:0:13716:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201232.110857:0:13716:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398ab0, osc lock ffff9023500dcd10, flags 0x400001002 00010000:00010000:2.0:1788201232.110908:0:13716:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e776ab58/0x4605c7c97e1c4aa5 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x418:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3145 expref: -99 pid: 13688 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201232.110924:0:13716:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e776ab58/0x4605c7c97e1c4aa5 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x418:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3145 expref: -99 pid: 13688 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788201232.110938:0:13716:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788201232.110942:0:13716:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e776ab58/0x4605c7c97e1c4aa5 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x418:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3145 expref: -99 pid: 13688 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201232.110950:0:13716:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e776ab58/0x4605c7c97e1c4aa5 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x418:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3145 expref: -99 pid: 13688 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1788201232.110971:0:13716:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788201232.112403:0:13716:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x3ef:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201232.112416:0:13716:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_release():245 00000020:00010000:2.0:1788201232.112421:0:13716:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201232.112426:0:13716:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398ab0 (ffff902350b9d758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201232.112433:0:13716:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201232.112436:0:13716:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398ab0 (ffff902350b9d638/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201232.112489:0:13716:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3ef:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201232.137254:0:13717:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201232.137293:0:13717:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201232.137318:0:13717:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:2.0:1788201232.137356:0:13717:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:2.0:1788201232.137365:0:13717:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201232.137374:0:13717:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201232.137378:0:13717:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c49e1 00000080:00010000:2.0:1788201232.137387:0:13717:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49e1 from it ffffa0b181cc3b58 00010000:00010000:2.0:1788201232.137392:0:13717:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:2.0:1788201232.137402:0:13717:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:2.0:1788201232.137428:0:13717:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201232.137438:0:13717:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102a.replay-single, flags=24 00000080:00200000:2.0:1788201232.137444:0:13717:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201232.137457:0:13717:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-7, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), flags=8 00000080:00200000:2.0:1788201232.137461:0:13717:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-7, dir=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), intent=getattr 00000002:00010000:2.0:1788201232.137492:0:13717:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-7,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3e9:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201232.137559:0:13717:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4b9a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 13717 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.137567:0:13717:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4b9a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13717 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.137579:0:13717:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4b9a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13717 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201232.137613:0:13717:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13717:1875064442629248:192.168.201.130@tcp:101:checkstat.0 00000100:00100000:2.0:1788201232.137969:0:13717:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448e40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201232.152629:0:13717:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13717:1875064442629248:192.168.201.130@tcp:101:checkstat.0 00010000:00010000:3.0:1788201232.152658:0:13717:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4b9a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33ec expref: -99 pid: 13717 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.152677:0:13717:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4b9a lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33ec expref: -99 pid: 13717 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.152686:0:13717:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4b9a lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33ec expref: -99 pid: 13717 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.152698:0:13717:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4b9a lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d33ec expref: -99 pid: 13717 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.152710:0:13717:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4b9a lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33ec expref: -99 pid: 13717 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.152725:0:13717:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4b9a lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d33ec expref: -99 pid: 13717 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.152734:0:13717:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4b9a lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d33ec expref: -99 pid: 13717 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.152741:0:13717:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4b9a lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d33ec expref: -99 pid: 13717 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201232.152753:0:13717:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c5500 x1875064442629248/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201248 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201232.152766:0:13717:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4b9a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33ec expref: -99 pid: 13717 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201232.152781:0:13717:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4b9a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33ec expref: -99 pid: 13717 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.152796:0:13717:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c4b5b lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3352 expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201232.152808:0:13717:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c4b5b lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3352 expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201232.152817:0:13717:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4b9a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33ec expref: -99 pid: 13717 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.152826:0:13717:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4b9a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d33ec expref: -99 pid: 13717 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.152840:0:13717:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4b9a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d33ec expref: -99 pid: 13717 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.152867:0:13717:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:3.0:1788201232.152917:0:13717:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3e9:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201232.153036:0:13717:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3f0:0x0](ffff9023449a9988): rc = 0 00000080:00200000:3.0:1788201232.153062:0:13717:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3f0:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201232.153078:0:13717:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f0:0x0](ffff9023449a9988) for lock 0x4605c7c97e1c4b5b 00000080:00010000:3.0:1788201232.153094:0:13717:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f0:0x0](ffff9023449a9988) 00000080:00010000:3.0:1788201232.153099:0:13717:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4b5b from it ffffa0b181cc3b20 00010000:00010000:3.0:1788201232.153123:0:13717:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c4b5b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3352 expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201232.153135:0:13717:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c4b5b lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f0:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3352 expref: -99 pid: 13712 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:3.0:1788201232.153178:0:13717:0:(file.c:6283:ll_getattr_dentry()) START file file-7:[0x200000bd1:0x3f0:0x0](ffff9023449a9988), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201232.153187:0:13717:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3f0:0x0](ffff9023449a9988),name=file-7 00000080:00200000:2.0:1788201232.180689:0:13718:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201232.180730:0:13718:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201232.180756:0:13718:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:2.0:1788201232.180780:0:13718:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:2.0:1788201232.180789:0:13718:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201232.180800:0:13718:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201232.180805:0:13718:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c49e1 00000080:00010000:2.0:1788201232.180816:0:13718:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49e1 from it ffffa0b181ccbb58 00010000:00010000:2.0:1788201232.180822:0:13718:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:2.0:1788201232.180832:0:13718:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:2.0:1788201232.180852:0:13718:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201232.180861:0:13718:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102a.replay-single, flags=8 00000080:00200008:2.0:1788201232.180871:0:13718:0:(file.c:6283:ll_getattr_dentry()) START file d102a.replay-single:[0x200000bd1:0x3e9:0x0](ffff9023449b1148), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201232.181461:0:13718:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148),name=d102a.replay-single 00000002:00010000:2.0:1788201232.181499:0:13718:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3e9:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201232.181509:0:13718:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 30 res 8589937617/1001 (0 0) 00010000:00010000:2.0:1788201232.181585:0:13718:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 13718 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.181595:0:13718:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13718 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201232.181607:0:13718:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13718 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201232.181641:0:13718:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13718:1875064442629504:192.168.201.130@tcp:101:rm.0 00000100:00100000:2.0:1788201232.181918:0:13718:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448f00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201232.189409:0:13718:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13718:1875064442629504:192.168.201.130@tcp:101:rm.0 00010000:00010000:3.0:1788201232.189448:0:13718:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33fa expref: -99 pid: 13718 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.189477:0:13718:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33fa expref: -99 pid: 13718 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.189488:0:13718:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33fa expref: -99 pid: 13718 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.189505:0:13718:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d33fa expref: -99 pid: 13718 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.189518:0:13718:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33fa expref: -99 pid: 13718 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.189534:0:13718:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d33fa expref: -99 pid: 13718 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.189543:0:13718:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d33fa expref: -99 pid: 13718 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.189550:0:13718:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d33fa expref: -99 pid: 13718 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201232.189564:0:13718:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c5180 x1875064442629504/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201248 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201232.189577:0:13718:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33fa expref: -99 pid: 13718 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.189588:0:13718:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1001 (0 0) 00000080:00200000:3.0:1788201232.189597:0:13718:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201232.189626:0:13718:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x3e9:0x0], updating i_size 7680 00000080:00010000:3.0:1788201232.189633:0:13718:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3e9:0x0](ffff9023449b1148) for lock 0x4605c7c97e1c4ba1 00000080:00010000:3.0:1788201232.189645:0:13718:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3e9:0x0](ffff9023449b1148) 00000080:00010000:3.0:1788201232.189649:0:13718:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4ba1 from it ffffa0b181ccbcb8 00010000:00010000:3.0:1788201232.189654:0:13718:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d33fa expref: -99 pid: 13718 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.189664:0:13718:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33fa expref: -99 pid: 13718 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201232.189797:0:13718:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201232.189818:0:13718:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201232.189832:0:13718:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:3.0:1788201232.189846:0:13718:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:3.0:1788201232.189853:0:13718:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201232.189859:0:13718:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201232.189863:0:13718:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c49e1 00000080:00010000:3.0:1788201232.189867:0:13718:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c49e1 from it ffffa0b181ccbbf8 00010000:00010000:3.0:1788201232.189870:0:13718:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:3.0:1788201232.189919:0:13718:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c49e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d2ec8 expref: -99 pid: 13662 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:3.0:1788201232.189930:0:13718:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201232.189938:0:13718:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102a.replay-single, flags=266 00000080:00200000:3.0:1788201232.189945:0:13718:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148), inode mode 41ed mask 44 00000080:00200008:3.0:1788201232.189956:0:13718:0:(file.c:1090:ll_file_open()) START file d102a.replay-single:[0x200000bd1:0x3e9:0x0](ffff9023449b1148), flags 704400 00000002:00010000:3.0:1788201232.189974:0:13718:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3e9:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:3.0:1788201232.190861:0:13718:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ba8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 13718 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.190872:0:13718:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ba8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13718 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201232.191151:0:13718:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201232.191165:0:13718:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352688a80 x1875064442629760/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201232.191175:0:13718:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ba8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13718 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201232.191205:0:13718:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13718:1875064442629760:192.168.201.130@tcp:101:rm.0 00000100:00100000:3.0:1788201232.191598:0:13718:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201232.203779:0:13718:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13718:1875064442629760:192.168.201.130@tcp:101:rm.0 00010000:00010000:3.0:1788201232.203813:0:13718:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ba8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 13718 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.203831:0:13718:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ba8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 13718 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.203839:0:13718:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ba8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 13718 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.203847:0:13718:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ba8 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 13718 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.203858:0:13718:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ba8 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 13718 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.203867:0:13718:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db1e00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201232.203872:0:13718:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ba8 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 13718 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.203924:0:13718:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ba8 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 13718 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.203933:0:13718:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ba8 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 13718 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.203942:0:13718:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201232.203950:0:13718:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ba8 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 13718 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201232.203957:0:13718:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ba8 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 13718 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201232.204251:0:13718:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff902352688a80 x1875064442629760/t8589938640(8589938640) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201248 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201232.204268:0:13718:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902352688a80 x1875064442629760/t8589938640(8589938640) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201248 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201232.204283:0:13718:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201232.204311:0:13718:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x3e9:0x0], updating i_size 7680 00000002:00100000:2.0:1788201232.204644:0:13718:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902352688a80 x1875064442629760/t8589938640(8589938640) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201248 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788201232.204912:0:13718:0:(file.c:6283:ll_getattr_dentry()) START file d102a.replay-single:[0x200000bd1:0x3e9:0x0](ffff9023449b1148), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201232.204922:0:13718:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148),name=d102a.replay-single 00000002:00010000:2.0:1788201232.204948:0:13718:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3e9:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201232.204969:0:13718:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33fa expref: -99 pid: 13718 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201232.205063:0:13718:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33fa expref: -99 pid: 13718 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201232.205074:0:13718:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201232.205083:0:13718:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3e9:0x0](ffff9023449b1148) 00000080:00010000:2.0:1788201232.205087:0:13718:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3e9:0x0](ffff9023449b1148) for lock 0x4605c7c97e1c4ba1 00000080:00010000:2.0:1788201232.205094:0:13718:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4ba1 from it ffffa0b181ccbcd8 00010000:00010000:2.0:1788201232.205098:0:13718:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d33fa expref: -99 pid: 13718 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201232.205110:0:13718:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33fa expref: -99 pid: 13718 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201232.205204:0:13718:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000bd1:0x3e9:0x0](ffff9023449b1148) pos/size0/7680 32bit_api 0 00000002:00010000:2.0:1788201232.205221:0:13718:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3e9:0x0]) in obj [0x200000bd1:0x3e9:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788201232.205228:0:13718:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33fa expref: -99 pid: 13718 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201232.205239:0:13718:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ba1 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3e9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d33fa expref: -99 pid: 13718 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1788201232.209494:0:13718:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4000 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13718:1875064442629888:192.168.201.130@tcp:37:rm.0 00000100:00100000:2.0:1788201232.209641:0:13718:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201234.226184:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201234.226232:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201234.226265:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458217600 00000100:00100000:3.0:1788201234.226275:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458217600 00000100:00100000:3.0:1788201234.226318:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458217728 00000100:00100000:3.0:1788201234.226321:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458217728 00000100:00100000:3.0:1788201234.226339:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1312 00000100:00100000:3.0:1788201234.226347:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902352688a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13440:x1875064458217600:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201234.226364:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c4a9e lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x418:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d313e expref: -99 pid: 13687 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201234.226441:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201234.226448:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902352688a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13440:x1875064458217600:12345-192.168.201.130@tcp:104: Request processed in 102us (276us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201234.226456:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1312 00000100:00100000:3.0:1788201234.226473:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1313 00000100:00100000:3.0:1788201234.226479:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902352688000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:12363:x1875064458217728:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201234.226489:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d90d9cc3/0x4605c7c97e1c4a66 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d309d expref: -99 pid: 13684 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201234.226519:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1788201234.226525:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902352688000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:12363:x1875064458217728:12345-192.168.201.130@tcp:104: Request processed in 46us (293us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201234.226530:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1313 00010000:00010000:2.0:1788201236.934336:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201238.978187:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201239.373104:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201239.373156:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201239.373179:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201239.373724:0:5591:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1788201239.373757:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458219264 00000100:00100000:3.0:1788201239.373772:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458219264 00000100:00100000:3.0:1788201239.373837:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458219392 00000100:00100000:3.0:1788201239.373840:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458219392 00000100:00100000:3.0:1788201239.373859:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1314 00000100:00100000:3.0:1788201239.373871:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c5880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13441:x1875064458219264:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201239.373928:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8ec2fff/0x4605c7c97e1c4a97 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d3137 expref: -99 pid: 13683 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201239.374039:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201239.374047:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c5880 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13441:x1875064458219264:12345-192.168.201.130@tcp:104: Request processed in 177us (954us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201239.374055:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1314 00000100:00100000:3.0:1788201239.374076:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1315 00000100:00100000:3.0:1788201239.374084:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023451ba050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13446:x1875064458219392:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201239.374093:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c4a82 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d30f8 expref: -99 pid: 13685 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201239.374122:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1788201239.374128:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023451ba050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13446:x1875064458219392:12345-192.168.201.130@tcp:104: Request processed in 46us (972us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201239.374135:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1315 00000100:00100000:2.0:1788201239.375046:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201239.375155:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201239.375194:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458220032 00000100:00100000:2.0:1788201239.375201:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458220032 00000100:00100000:2.0:1788201239.375234:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458219904 00000100:00100000:2.0:1788201239.375237:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458219904 00000100:00100000:2.0:1788201239.375254:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1316 00000100:00100000:2.0:1788201239.375263:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c4e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13448:x1875064458220032:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201239.375279:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375334:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to priority blp list, 3 blwis in pool 00000100:00100000:2.0:1788201239.375341:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c4e00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13448:x1875064458220032:12345-192.168.201.130@tcp:104: Request processed in 79us (300us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201239.375348:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1316 00000100:00100000:2.0:1788201239.375363:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1317 00000100:00100000:2.0:1788201239.375369:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c6300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6705:x1875064458219904:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201239.375377:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000035fa0f84/0x4605c7c97e1c4b1c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d323a expref: -99 pid: 13689 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375406:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/4 locks to priority blp list, 4 blwis in pool 00000100:00100000:2.0:1788201239.375412:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c6300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:6705:x1875064458219904:12345-192.168.201.130@tcp:104: Request processed in 43us (257us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201239.375419:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1317 00010000:00010000:2.0:1788201239.375461:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 3 total in blp. (3 blwis in pool) 00010000:00010000:2.0:1788201239.375469:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8ec2fff/0x4605c7c97e1c4a97 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d3137 expref: -99 pid: 13683 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375480:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839d600 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201239.375487:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8ec2fff/0x4605c7c97e1c4a97 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d3137 expref: -99 pid: 13683 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375591:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8ec2fff/0x4605c7c97e1c4a97 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3137 expref: -99 pid: 13683 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375600:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201239.375626:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8ec2fff/0x4605c7c97e1c4a97 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3137 expref: -99 pid: 13683 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375636:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8ec2fff/0x4605c7c97e1c4a97 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x416:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3137 expref: -99 pid: 13683 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375665:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:2.0:1788201239.375670:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c4a82 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d30f8 expref: -99 pid: 13685 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375678:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839ce00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201239.375682:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c4a82 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d30f8 expref: -99 pid: 13685 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375723:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c4a82 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d30f8 expref: -99 pid: 13685 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375732:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201239.375738:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c4a82 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d30f8 expref: -99 pid: 13685 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375746:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c4a82 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d30f8 expref: -99 pid: 13685 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375761:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1788201239.375765:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375773:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351685a00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201239.375778:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375812:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375820:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201239.375825:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375840:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ac5a9f7d/0x4605c7c97e1c4ad6 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x417:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d314c expref: -99 pid: 13686 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375858:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201239.375861:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000035fa0f84/0x4605c7c97e1c4b1c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d323a expref: -99 pid: 13689 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.375869:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351684c00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201239.375873:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000035fa0f84/0x4605c7c97e1c4b1c lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d323a expref: -99 pid: 13689 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.376131:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000035fa0f84/0x4605c7c97e1c4b1c lrc: 2/0,0 mode: --/PR res: [0x240000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d323a expref: -99 pid: 13689 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.376140:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201239.376146:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000035fa0f84/0x4605c7c97e1c4b1c lrc: 1/0,0 mode: --/PR res: [0x240000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d323a expref: -99 pid: 13689 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.376154:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000035fa0f84/0x4605c7c97e1c4b1c lrc: 0/0,0 mode: --/PR res: [0x240000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d323a expref: -99 pid: 13689 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.376168:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201239.376170:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201239.376172:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201239.376200:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442633728:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201239.376281:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442633984:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201239.376309:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788201239.376315:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442633856:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201239.376334:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442634112:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201239.380377:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201239.381738:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201239.381750:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e776ab58/0x4605c7c97e1c4aa5 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x418:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d3145 expref: -99 pid: 13688 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.381763:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345803600 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201239.381827:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e776ab58/0x4605c7c97e1c4aa5 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x418:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d3145 expref: -99 pid: 13688 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.382045:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e776ab58/0x4605c7c97e1c4aa5 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x418:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3145 expref: -99 pid: 13688 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.382055:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201239.382084:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e776ab58/0x4605c7c97e1c4aa5 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x418:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3145 expref: -99 pid: 13688 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.382094:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e776ab58/0x4605c7c97e1c4aa5 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x418:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3145 expref: -99 pid: 13688 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201239.382129:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201239.382132:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201239.382136:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201239.382148:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201239.382170:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442634240:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201239.387682:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442633856:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201239.387703:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442634112:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201239.387711:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442633728:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201239.387722:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442633984:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201239.391916:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442634240:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201240.427231:0:14284:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345831500 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14284:1875064442634368:192.168.201.130@tcp:101:lfs.0 00010000:00010000:3.0:1788201240.427260:0:14284:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.427274:0:14284:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.427285:0:14284:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.427296:0:14284:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.427310:0:14284:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.427318:0:14284:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.427326:0:14284:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201240.427337:0:14284:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902345831500 x1875064442634368/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201256 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201240.427350:0:14284:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.427361:0:14284:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201240.427368:0:14284:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201240.427401:0:14284:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:3.0:1788201240.427406:0:14284:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bb6 00000080:00010000:3.0:1788201240.427411:0:14284:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201240.427414:0:14284:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bb6 from it ffffa0b181d0bb58 00010000:00010000:3.0:1788201240.427417:0:14284:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.427426:0:14284:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201240.427457:0:14284:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201240.427485:0:14284:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d102b.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=13 00000080:00200000:3.0:1788201240.427489:0:14284:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d102b.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=getattr 00000002:00010000:3.0:1788201240.427510:0:14284:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d102b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201240.427552:0:14284:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bbd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 14284 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.427560:0:14284:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bbd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14284 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.427569:0:14284:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bbd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14284 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201240.427586:0:14284:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14284:1875064442634496:192.168.201.130@tcp:101:lfs.0 00000100:00100000:3.0:1788201240.427667:0:14284:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201240.437088:0:14284:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14284:1875064442634496:192.168.201.130@tcp:101:lfs.0 00010000:00010000:3.0:1788201240.437119:0:14284:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bbd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14284 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.437134:0:14284:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bbd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 14284 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.437140:0:14284:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bbd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 14284 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.437147:0:14284:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bbd lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 14284 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.437159:0:14284:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bbd lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 14284 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.437165:0:14284:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db1600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201240.437170:0:14284:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bbd lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 14284 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.437177:0:14284:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bbd lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 14284 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.437183:0:14284:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bbd lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 14284 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.437191:0:14284:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201240.437197:0:14284:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bbd lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 14284 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.437203:0:14284:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bbd lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 14284 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201240.437220:0:14284:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff90235268a680 x1875064442634496/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201256 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201240.437235:0:14284:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1788201240.437263:0:14284:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201240.437277:0:14284:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201240.437286:0:14284:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bb6 from it ffffa0b181d0b930 00010000:00010000:3.0:1788201240.437291:0:14284:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201240.437299:0:14284:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201240.437375:0:14284:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201240.437390:0:14284:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201240.437397:0:14284:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201240.437405:0:14284:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201240.437410:0:14284:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201240.437415:0:14284:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201240.437418:0:14284:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bb6 00000080:00010000:3.0:1788201240.437423:0:14284:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bb6 from it ffffa0b181d0bbe8 00010000:00010000:3.0:1788201240.437426:0:14284:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201240.437432:0:14284:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201240.437440:0:14284:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201240.437448:0:14284:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102b.replay-single, flags=16392 00000080:00200008:3.0:1788201240.437470:0:14284:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788201240.437475:0:14284:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201240.437482:0:14284:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201240.437488:0:14284:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201240.437496:0:14284:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201240.437502:0:14284:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201240.437506:0:14284:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201240.437508:0:14284:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bb6 00000080:00010000:3.0:1788201240.437511:0:14284:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bb6 from it ffffa0b181d0bcb8 00010000:00010000:3.0:1788201240.437515:0:14284:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201240.437523:0:14284:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1788201240.438152:0:14284:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788201240.438157:0:14284:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201240.438166:0:14284:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201240.438174:0:14284:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201240.438185:0:14284:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788201240.438191:0:14284:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201240.438195:0:14284:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201240.438198:0:14284:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bb6 00000080:00010000:3.0:1788201240.438202:0:14284:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bb6 from it ffffa0b181d0bcb8 00010000:00010000:3.0:1788201240.438206:0:14284:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201240.438213:0:14284:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201240.438334:0:14284:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201240.438342:0:14284:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201240.438350:0:14284:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201240.438358:0:14284:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201240.438363:0:14284:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201240.438367:0:14284:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201240.438369:0:14284:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bb6 00000080:00010000:3.0:1788201240.438373:0:14284:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bb6 from it ffffa0b181d0bbb0 00010000:00010000:3.0:1788201240.438376:0:14284:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201240.438383:0:14284:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788201240.438391:0:14284:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 44 00000080:00200008:3.0:1788201240.438398:0:14284:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 300000 00000080:00200080:3.0:1788201240.438414:0:14284:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c03066f0 arg=7ffd1d44ec70 00000080:00200000:3.0:1788201240.438426:0:14284:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) name=d102b.replay-single stripe_offset=-1 stripe_count=1, hash_type=0 00010000:00010000:3.0:1788201240.438455:0:14284:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788201240.438517:0:14284:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788201240.438524:0:14284:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201240.438527:0:14284:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4bb6 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3535 expref: -99 pid: 14284 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000020:00100000:3.0:1788201240.438550:0:14284:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201240.438561:0:14284:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90235268a680 x1875064442634752/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 568/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201240.438584:0:14284:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14284:1875064442634752:192.168.201.130@tcp:36:lfs.0 00000100:00100000:3.0:1788201240.438692:0:14284:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201240.448600:0:14284:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14284:1875064442634752:192.168.201.130@tcp:36:lfs.0 00000080:00200000:3.0:1788201240.448657:0:14284:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448) 00000080:00200000:3.0:1788201240.448665:0:14284:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x3f1:0x0], updating i_size 512 00000080:00200008:3.0:1788201240.448700:0:14284:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 300000 00000100:00100000:3.0:1788201240.505217:0:14290:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345833b80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14290:1875064442634880:192.168.201.130@tcp:101:lfs.0 00010000:00010000:3.0:1788201240.505251:0:14290:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.505266:0:14290:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.505283:0:14290:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.505295:0:14290:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.505308:0:14290:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.505316:0:14290:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.505323:0:14290:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201240.505335:0:14290:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902345833b80 x1875064442634880/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201256 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201240.505350:0:14290:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.505362:0:14290:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201240.505369:0:14290:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201240.505403:0:14290:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:3.0:1788201240.505409:0:14290:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bc4 00000080:00010000:3.0:1788201240.505414:0:14290:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201240.505417:0:14290:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bc4 from it ffffa0b181ce3bf8 00010000:00010000:3.0:1788201240.505421:0:14290:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.505429:0:14290:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201240.505455:0:14290:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201240.505495:0:14290:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=d102b.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), file ffff902343d62480, open_flags 18800, mode 0 opened 0 00000080:00200000:3.0:1788201240.505502:0:14290:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d102b.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=open 00000002:00010000:3.0:1788201240.505525:0:14290:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d102b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0304001 00010000:00010000:3.0:1788201240.505581:0:14290:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bcb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 14290 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.505589:0:14290:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bcb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14290 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201240.505603:0:14290:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201240.505612:0:14290:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352689180 x1875064442635136/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201240.505621:0:14290:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bcb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14290 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201240.505644:0:14290:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14290:1875064442635136:192.168.201.130@tcp:101:lfs.0 00000100:00100000:3.0:1788201240.506008:0:14290:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201240.518833:0:14290:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14290:1875064442635136:192.168.201.130@tcp:101:lfs.0 00010000:00010000:3.0:1788201240.519121:0:14290:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bcb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14290 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.519489:0:14290:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bcb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 14290 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.519501:0:14290:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bcb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 14290 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.519511:0:14290:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bcb lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 14290 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.519521:0:14290:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bcb lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 14290 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.519529:0:14290:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db1600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201240.519534:0:14290:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bcb lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 14290 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.519543:0:14290:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bcb lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 14290 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.519551:0:14290:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bcb lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 14290 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.519559:0:14290:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201240.519567:0:14290:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bcb lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 14290 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.519575:0:14290:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c4bcb lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 14290 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201240.519600:0:14290:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff902352689180 x1875064442635136/t8589938653(8589938653) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/608 e 0 to 0 dl 1788201256 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201240.519622:0:14290:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902352689180 x1875064442635136/t8589938653(8589938653) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/608 e 0 to 0 dl 1788201256 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201240.519638:0:14290:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201240.519666:0:14290:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x3f1:0x0], updating i_size 512 00000080:00200000:3.0:1788201240.519669:0:14290:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3f1:0x0](ffff90234497f448): rc = 0 00000080:00010000:3.0:1788201240.519675:0:14290:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f1:0x0](ffff90234497f448) for lock 0x0 00000080:00200008:3.0:1788201240.519702:0:14290:0:(file.c:1090:ll_file_open()) START file d102b.replay-single:[0x200000bd1:0x3f1:0x0](ffff90234497f448), flags 304000 00000002:00100000:3.0:1788201240.519868:0:14290:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902352689180 x1875064442635136/t8589938653(8589938653) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/608 e 0 to 0 dl 1788201256 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201240.519985:0:14290:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448), inode mode 41ed mask 44 00000080:00200000:3.0:1788201240.520061:0:14290:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201240.520083:0:14290:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201240.520102:0:14290:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201240.520119:0:14290:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201240.520128:0:14290:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201240.520135:0:14290:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201240.520139:0:14290:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bc4 00000080:00010000:3.0:1788201240.520327:0:14290:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bc4 from it ffffa0b181ce3bf8 00010000:00010000:3.0:1788201240.520333:0:14290:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201240.520344:0:14290:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201240.520432:0:14290:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201240.520453:0:14290:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=d102b.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), file ffff902344d6ef00, open_flags 28800, mode 0 opened 0 00000080:00200000:3.0:1788201240.520459:0:14290:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d102b.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=open 00000002:00010000:3.0:1788201240.520478:0:14290:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d102b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0504001 00010000:00010000:3.0:1788201240.520538:0:14290:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4bd2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 14290 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.520546:0:14290:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4bd2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14290 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201240.520560:0:14290:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201240.520568:0:14290:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352688a80 x1875064442635392/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201240.520577:0:14290:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4bd2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14290 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201240.520604:0:14290:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14290:1875064442635392:192.168.201.130@tcp:101:lfs.0 00000100:00100000:3.0:1788201240.521088:0:14290:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201240.538618:0:14290:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345830700 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14290:1875064442635776:192.168.201.130@tcp:33:lfs.0 00000080:00200080:3.0:1788201240.538701:0:14290:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448) cmd=800466af arg=7ffef5c0aa20 00000080:00200008:3.0:1788201240.539168:0:14290:0:(file.c:479:ll_file_release()) START file d102b.replay-single:[0x200000bd1:0x3f1:0x0](ffff90234497f448), flags 504000 00000002:00100000:3.0:1788201240.539245:0:14290:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff902352689180 x1875064442635136/t8589938653(8589938653) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/608 e 0 to 0 dl 1788201256 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201240.539263:0:14290:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201240.539272:0:14290:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352688000 x1875064442636032/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201240.539302:0:14290:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14290:1875064442636032:192.168.201.130@tcp:35:lfs.0 00000100:00100000:3.0:1788201240.539591:0:14290:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201240.541586:0:14290:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14290:1875064442636032:192.168.201.130@tcp:35:lfs.0 00000002:00100000:3.0:1788201240.541628:0:14290:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff902352689180 x1875064442635136/t8589938653(8589938653) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/608 e 0 to 0 dl 1788201256 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201240.590420:0:14295:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201240.590454:0:14295:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201240.590474:0:14295:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201240.590494:0:14295:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201240.590502:0:14295:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201240.590510:0:14295:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201240.590514:0:14295:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bc4 00000080:00010000:3.0:1788201240.590522:0:14295:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bc4 from it ffffa0b181523bf8 00010000:00010000:3.0:1788201240.590526:0:14295:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201240.590536:0:14295:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201240.590553:0:14295:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201240.590566:0:14295:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d102b.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=25 00000080:00200000:3.0:1788201240.590572:0:14295:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d102b.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=0 00000002:00010000:3.0:1788201240.590584:0:14295:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d102b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1788201240.590634:0:14295:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4bd9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 14295 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.590642:0:14295:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4bd9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14295 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.590653:0:14295:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4bd9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14295 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201240.590683:0:14295:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14295:1875064442636160:192.168.201.130@tcp:101:touch.0 00000100:00100000:3.0:1788201240.590800:0:14295:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201240.594359:0:14295:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14295:1875064442636160:192.168.201.130@tcp:101:touch.0 00010000:00010000:3.0:1788201240.594391:0:14295:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4bd9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35c1 expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.594413:0:14295:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4bd9 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35c1 expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.594422:0:14295:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4bd9 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35c1 expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.594434:0:14295:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4bd9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d35c1 expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.594447:0:14295:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4bd9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35c1 expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.594468:0:14295:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4bd9 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d35c1 expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.594477:0:14295:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4bd9 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d35c1 expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.594485:0:14295:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4bd9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d35c1 expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201240.594501:0:14295:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff902352688000 x1875064442636160/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201256 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201240.594515:0:14295:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4bd9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35c1 expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.594526:0:14295:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1009 (0 0) 00000080:00200000:3.0:1788201240.594534:0:14295:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201240.594562:0:14295:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x3f1:0x0], updating i_size 512 00000080:00200000:3.0:1788201240.594566:0:14295:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3f1:0x0](ffff90234497f448): rc = 0 00000080:00010000:3.0:1788201240.594573:0:14295:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f1:0x0](ffff90234497f448) for lock 0x4605c7c97e1c4bd9 00000080:00010000:3.0:1788201240.594587:0:14295:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f1:0x0](ffff90234497f448) 00000080:00010000:3.0:1788201240.594592:0:14295:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bd9 from it ffffa0b181523ac8 00010000:00010000:3.0:1788201240.594595:0:14295:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4bd9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d35c1 expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.594604:0:14295:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c4bd9 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35c1 expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201240.594651:0:14295:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448), inode mode 41ed mask 1 00000080:00200000:3.0:1788201240.594674:0:14295:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448), inode mode 41ed mask 3 00000080:00200000:3.0:1788201240.594680:0:14295:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file-1, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), file ffff902344d6f500, open_flags 8941, mode 81b6 opened 0 00000080:00200000:3.0:1788201240.594688:0:14295:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-1, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), intent=open|creat 00000002:00010000:3.0:1788201240.594712:0:14295:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-1,[0x200000bd1:0x3f2:0x0]) in obj [0x200000bd1:0x3f1:0x0], intent: open|creat flags 0104502 00010000:00010000:3.0:1788201240.594724:0:14295:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4bd2 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d359e expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201240.594811:0:14295:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4bd2 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d359e expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201240.594818:0:14295:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201240.594832:0:14295:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4bd2 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d359e expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201240.594866:0:14295:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 14295 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.594873:0:14295:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14295 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201240.595339:0:14295:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201240.595350:0:14295:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352688000 x1875064442636416/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201240.595360:0:14295:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14295 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201240.595387:0:14295:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14295:1875064442636416:192.168.201.130@tcp:101:touch.0 00000100:00100000:3.0:1788201240.595536:0:14295:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201240.600043:0:14295:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14295:1875064442636416:192.168.201.130@tcp:101:touch.0 00010000:00010000:3.0:1788201240.600072:0:14295:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35dd expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.600094:0:14295:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35dd expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.600103:0:14295:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35dd expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.600114:0:14295:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d35dd expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.600126:0:14295:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35dd expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.600140:0:14295:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d35dd expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.600148:0:14295:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d35dd expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.600155:0:14295:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d35dd expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201240.600447:0:14295:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff902352688000 x1875064442636416/t8589938657(8589938657) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201256 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201240.600466:0:14295:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902352688000 x1875064442636416/t8589938657(8589938657) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201256 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201240.600483:0:14295:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35dd expref: -99 pid: 14295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201240.600498:0:14295:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35dd expref: -99 pid: 14295 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.600509:0:14295:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1010 (0 0) 00000080:00200000:3.0:1788201240.600516:0:14295:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x3f1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201240.600529:0:14295:0:(namei.c:1628:ll_create_it()) VFS Op:name=file-1, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), intent=open|creat 00000080:00200000:3.0:1788201240.600558:0:14295:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x3f2:0x0](ffff90234497d348) 00000080:00200000:3.0:1788201240.600681:0:14295:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3f2:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201240.600691:0:14295:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x3f2:0x0](ffff90234497d348) 00000080:00010000:3.0:1788201240.600694:0:14295:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f2:0x0](ffff90234497d348) for lock 0x4605c7c97e1c4be0 00000080:00010000:3.0:1788201240.600703:0:14295:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f2:0x0](ffff90234497d348) 00000080:00200008:3.0:1788201240.600717:0:14295:0:(file.c:1090:ll_file_open()) START file file-1:[0x200000bd1:0x3f2:0x0](ffff90234497d348), flags 104501 00000002:00100000:3.0:1788201240.600735:0:14295:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902352688000 x1875064442636416/t8589938657(8589938657) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201256 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201240.600773:0:14295:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4be0 from it ffff9023533eaa20 00010000:00010000:3.0:1788201240.600779:0:14295:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d35dd expref: -99 pid: 14295 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.600794:0:14295:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d35dd expref: -99 pid: 14295 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201240.600818:0:14295:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f2:0x0](ffff90234497d348), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201240.600972:0:14295:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-1:[0x200000bd1:0x3f2:0x0](ffff90234497d348) current size 0, valid attrs 20070, mode 9023, uid -2125316528, gid -24399, new size 18446744071973566999, atime 1788201240.000000000, mtime 1788201240.000000000, ctime 1788201240.000000000 00010000:00010000:3.0:1788201240.600996:0:14295:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d35dd expref: -99 pid: 14295 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201240.601024:0:14295:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3f2:0x0]: losing layout lock 00010000:00010000:3.0:1788201240.601070:0:14295:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d35dd expref: -99 pid: 14295 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201240.601077:0:14295:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201240.601080:0:14295:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4be0 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d35dd expref: -99 pid: 14295 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788201240.601104:0:14295:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201240.601111:0:14295:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352689180 x1875064442636672/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201240.601134:0:14295:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14295:1875064442636672:192.168.201.130@tcp:36:touch.0 00000100:00100000:3.0:1788201240.601294:0:14295:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201240.606362:0:14295:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14295:1875064442636672:192.168.201.130@tcp:36:touch.0 00000080:00200000:3.0:1788201240.606471:0:14295:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3f2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:3.0:1788201240.606496:0:14295:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x3f2:0x0](ffff90234497d348) 00000002:00010000:3.0:1788201240.606510:0:14295:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3f2:0x0]) in obj [0x200000bd1:0x3f2:0x0], intent: layout flags 0 00010000:00010000:3.0:1788201240.608253:0:14295:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4be7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 14295 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.608275:0:14295:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4be7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 14295 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.608296:0:14295:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4be7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 14295 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201240.608341:0:14295:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14295:1875064442636800:192.168.201.130@tcp:101:touch.0 00000100:00100000:3.0:1788201240.608467:0:14295:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201240.611693:0:14295:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14295:1875064442636800:192.168.201.130@tcp:101:touch.0 00010000:00010000:3.0:1788201240.611734:0:14295:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4be7 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d35f2 expref: -99 pid: 14295 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.611754:0:14295:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4be7 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35f2 expref: -99 pid: 14295 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201240.611771:0:14295:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff90235268aa00 x1875064442636800/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201256 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201240.611784:0:14295:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4be7 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35f2 expref: -99 pid: 14295 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201240.611799:0:14295:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4be7 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35f2 expref: -99 pid: 14295 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.611814:0:14295:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1010 (0 0) 00000080:00200000:3.0:1788201240.611824:0:14295:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x3f2:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1788201240.611846:0:14295:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f2:0x0](ffff90234497d348) for lock 0x4605c7c97e1c4be7 00000080:00010000:3.0:1788201240.611869:0:14295:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4be7 from it ffffa0b181523a78 00010000:00010000:3.0:1788201240.611874:0:14295:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4be7 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d35f2 expref: -99 pid: 14295 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.611924:0:14295:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4be7 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35f2 expref: -99 pid: 14295 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201240.611950:0:14295:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4be7 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35f2 expref: -99 pid: 14295 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201240.611961:0:14295:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4be7 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35f2 expref: -99 pid: 14295 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201240.611971:0:14295:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x3f2:0x0](00000000e95b4195) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4be7 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d35f2 expref: -99 pid: 14295 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201240.612067:0:14295:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3f2:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:3.0:1788201240.612086:0:14295:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4be7 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d35f2 expref: -99 pid: 14295 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201240.612092:0:14295:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c4be7 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3f2:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d35f2 expref: -99 pid: 14295 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:3.0:1788201240.612114:0:14295:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1788201240.612290:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442636928:192.168.201.130@tcp:2:touch.0 00000100:00100000:3.0:1788201240.615013:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442636928:192.168.201.130@tcp:2:touch.0 00000080:00200000:2.0:1788201240.684180:0:14296:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f3:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201240.684203:0:14296:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-2:[0x200000bd1:0x3f3:0x0](ffff9023449a3248) current size 0, valid attrs 100201f0, mode 9023, uid -2117747120, gid -24399, new size 18446744071973566999, atime 1788201240.000000000, mtime 1788201240.000000000, ctime 1788201240.000000000, rc 0 00000080:00200008:2.0:1788201240.684234:0:14296:0:(file.c:479:ll_file_release()) START file file-2:[0x200000bd1:0x3f3:0x0](ffff9023449a3248), flags 104001 00000002:00100000:2.0:1788201240.684307:0:14296:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff902345830a80 x1875064442637440/t8589938660(8589938660) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201256 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201240.684320:0:14296:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201240.684327:0:14296:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c6680 x1875064442638208/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201240.684353:0:14296:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14296:1875064442638208:192.168.201.130@tcp:35:touch.0 00000100:00100000:2.0:1788201240.684450:0:14296:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bb40 going to sleep for 11 seconds 00000002:00010000:3.0:1788201240.815614:0:14298:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c4c0a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3670 expref: -99 pid: 14298 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.815629:0:14298:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1013 (0 0) 00000080:00200000:3.0:1788201240.815636:0:14298:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x3f1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201240.815650:0:14298:0:(namei.c:1628:ll_create_it()) VFS Op:name=file-4, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), intent=open|creat 00000080:00200000:3.0:1788201240.815688:0:14298:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x3f5:0x0](ffff90234497aa08) 00000080:00200000:3.0:1788201240.815832:0:14298:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3f5:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201240.815845:0:14298:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x3f5:0x0](ffff90234497aa08) 00000080:00010000:3.0:1788201240.815849:0:14298:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f5:0x0](ffff90234497aa08) for lock 0x4605c7c97e1c4c0a 00000080:00010000:3.0:1788201240.815866:0:14298:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f5:0x0](ffff90234497aa08) 00000080:00200008:3.0:1788201240.815901:0:14298:0:(file.c:1090:ll_file_open()) START file file-4:[0x200000bd1:0x3f5:0x0](ffff90234497aa08), flags 104501 00000002:00100000:3.0:1788201240.815924:0:14298:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902345832d80 x1875064442639488/t8589938666(8589938666) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201256 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201240.816097:0:14298:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4c0a from it ffff9023450e7900 00010000:00010000:3.0:1788201240.816103:0:14298:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c4c0a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3670 expref: -99 pid: 14298 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.816113:0:14298:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c4c0a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3670 expref: -99 pid: 14298 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201240.816136:0:14298:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f5:0x0](ffff90234497aa08), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201240.816175:0:14298:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-4:[0x200000bd1:0x3f5:0x0](ffff90234497aa08) current size 0, valid attrs 20070, mode 9023, uid -2123448752, gid -24399, new size 18446744071973566999, atime 1788201240.000000000, mtime 1788201240.000000000, ctime 1788201240.000000000 00010000:00010000:3.0:1788201240.816196:0:14298:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c4c0a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d3670 expref: -99 pid: 14298 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201240.816220:0:14298:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3f5:0x0]: losing layout lock 00010000:00010000:3.0:1788201240.816264:0:14298:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c4c0a lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3670 expref: -99 pid: 14298 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201240.816272:0:14298:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201240.816275:0:14298:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c4c0a lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3670 expref: -99 pid: 14298 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788201240.816301:0:14298:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201240.816308:0:14298:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90235268a300 x1875064442639744/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201240.816330:0:14298:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14298:1875064442639744:192.168.201.130@tcp:36:touch.0 00000100:00100000:3.0:1788201240.816739:0:14298:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6840 going to sleep for 11 seconds 00000080:00200000:3.0:1788201240.833995:0:14298:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f5:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201240.834020:0:14298:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-4:[0x200000bd1:0x3f5:0x0](ffff90234497aa08) current size 0, valid attrs 100201f0, mode 9023, uid -2123448752, gid -24399, new size 18446744071973566999, atime 1788201240.000000000, mtime 1788201240.000000000, ctime 1788201240.000000000, rc 0 00000080:00200008:3.0:1788201240.834055:0:14298:0:(file.c:479:ll_file_release()) START file file-4:[0x200000bd1:0x3f5:0x0](ffff90234497aa08), flags 104001 00000002:00100000:3.0:1788201240.834125:0:14298:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff902345832d80 x1875064442639488/t8589938666(8589938666) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201256 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201240.834138:0:14298:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201240.834143:0:14298:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352689f80 x1875064442640256/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201240.834169:0:14298:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689f80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14298:1875064442640256:192.168.201.130@tcp:35:touch.0 00000100:00100000:3.0:1788201240.834267:0:14298:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6900 going to sleep for 11 seconds 00000080:00200000:3.0:1788201240.886272:0:14299:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201240.886308:0:14299:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-5:[0x200000bd1:0x3f6:0x0](ffff9023449a42c8) current size 0, valid attrs 100201f0, mode 9023, uid -2117714352, gid -24399, new size 18446744071973566999, atime 1788201240.000000000, mtime 1788201240.000000000, ctime 1788201240.000000000, rc 0 00000080:00200008:3.0:1788201240.886354:0:14299:0:(file.c:479:ll_file_release()) START file file-5:[0x200000bd1:0x3f6:0x0](ffff9023449a42c8), flags 104001 00000002:00100000:3.0:1788201240.886439:0:14299:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b1c00 x1875064442640512/t8589938669(8589938669) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201256 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201240.886454:0:14299:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201240.886460:0:14299:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352688e00 x1875064442641280/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201240.886488:0:14299:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14299:1875064442641280:192.168.201.130@tcp:35:touch.0 00000100:00100000:3.0:1788201240.886595:0:14299:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6000 going to sleep for 11 seconds 00000100:00100000:3.0:1788201240.890464:0:14299:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14299:1875064442641280:192.168.201.130@tcp:35:touch.0 00000002:00100000:3.0:1788201240.890509:0:14299:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b1c00 x1875064442640512/t8589938669(8589938669) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201256 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201240.969012:0:14301:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3f8:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:2.0:1788201240.969047:0:14301:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x3f8:0x0](ffff9023449b4b08) 00000002:00010000:2.0:1788201240.969064:0:14301:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3f8:0x0]) in obj [0x200000bd1:0x3f8:0x0], intent: layout flags 0 00010000:00010000:2.0:1788201240.969138:0:14301:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4c3b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 14301 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201240.969150:0:14301:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4c3b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 14301 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201240.969162:0:14301:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4c3b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 14301 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201240.969190:0:14301:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14301:1875064442642944:192.168.201.130@tcp:101:touch.0 00000100:00100000:2.0:1788201240.969397:0:14301:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b540 going to sleep for 11 seconds 00000100:00100000:3.0:1788201240.973517:0:14301:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14301:1875064442642944:192.168.201.130@tcp:101:touch.0 00010000:00010000:3.0:1788201240.973563:0:14301:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4c3b lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3718 expref: -99 pid: 14301 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.973585:0:14301:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4c3b lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3718 expref: -99 pid: 14301 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201240.973606:0:14301:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9023458c7800 x1875064442642944/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201256 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201240.973621:0:14301:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4c3b lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3718 expref: -99 pid: 14301 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201240.973637:0:14301:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4c3b lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3718 expref: -99 pid: 14301 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.973656:0:14301:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1016 (0 0) 00000080:00200000:3.0:1788201240.973664:0:14301:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x3f8:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1788201240.973688:0:14301:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f8:0x0](ffff9023449b4b08) for lock 0x4605c7c97e1c4c3b 00000080:00010000:3.0:1788201240.973698:0:14301:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4c3b from it ffffa0b181c63a78 00010000:00010000:3.0:1788201240.973703:0:14301:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4c3b lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3718 expref: -99 pid: 14301 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201240.973716:0:14301:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4c3b lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3718 expref: -99 pid: 14301 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201240.973758:0:14301:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4c3b lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3718 expref: -99 pid: 14301 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201240.973772:0:14301:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4c3b lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3718 expref: -99 pid: 14301 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201240.973780:0:14301:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x3f8:0x0](00000000d67d3de0) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4c3b lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3718 expref: -99 pid: 14301 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201240.973911:0:14301:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3f8:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:3.0:1788201240.973938:0:14301:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4c3b lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3718 expref: -99 pid: 14301 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201240.973945:0:14301:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c4c3b lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3718 expref: -99 pid: 14301 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:3.0:1788201240.973982:0:14301:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201240.974093:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201240.974113:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442643072:192.168.201.130@tcp:2:touch.0 00000100:00100000:3.0:1788201240.987821:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442643072:192.168.201.130@tcp:2:touch.0 00000080:00200000:2.0:1788201240.988041:0:14301:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f8:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201240.988078:0:14301:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-7:[0x200000bd1:0x3f8:0x0](ffff9023449b4b08) current size 0, valid attrs 100201f0, mode 9023, uid -2117714352, gid -24399, new size 18446744071973566999, atime 1788201240.000000000, mtime 1788201240.000000000, ctime 1788201240.000000000, rc 0 00000080:00200008:2.0:1788201240.988118:0:14301:0:(file.c:479:ll_file_release()) START file file-7:[0x200000bd1:0x3f8:0x0](ffff9023449b4b08), flags 104001 00000002:00100000:2.0:1788201240.988200:0:14301:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b3800 x1875064442642560/t8589938675(8589938675) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201256 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201240.988214:0:14301:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201240.988220:0:14301:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c7b80 x1875064442643328/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201240.988248:0:14301:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14301:1875064442643328:192.168.201.130@tcp:35:touch.0 00000100:00100000:2.0:1788201240.988354:0:14301:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201241.000465:0:14301:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14301:1875064442643328:192.168.201.130@tcp:35:touch.0 00000002:00100000:3.0:1788201241.000519:0:14301:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b3800 x1875064442642560/t8589938675(8589938675) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201256 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201241.990420:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442643584:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201241.990433:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442643456:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201241.990573:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442643712:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201241.994397:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442643840:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201242.018794:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442643584:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201242.018863:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442643456:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201242.021869:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442643840:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201242.021950:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442643712:192.168.201.130@tcp:400:kworker.0 00010000:00010000:3.0:1788201242.049191:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788201242.102915:0:14313:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201242.102972:0:14313:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201242.102997:0:14313:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788201242.103016:0:14313:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788201242.103025:0:14313:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201242.103034:0:14313:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201242.103038:0:14313:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bc4 00000080:00010000:2.0:1788201242.103048:0:14313:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bc4 from it ffffa0b181bebb58 00010000:00010000:2.0:1788201242.103052:0:14313:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788201242.103061:0:14313:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788201242.103082:0:14313:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201242.103092:0:14313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102b.replay-single, flags=25 00000080:00200000:2.0:1788201242.103098:0:14313:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448), inode mode 41ed mask 1 00000080:00200000:2.0:1788201242.103111:0:14313:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-2, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), flags=9 00000080:00200000:2.0:1788201242.106206:0:14313:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-2, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), intent=getattr 00000002:00010000:2.0:1788201242.106299:0:14313:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-2,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3f1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201242.106387:0:14313:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 14313 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.106405:0:14313:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 14313 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.106418:0:14313:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 14313 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201242.106451:0:14313:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6d80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14313:1875064442644096:192.168.201.130@tcp:101:chmod.0 00000100:00100000:2.0:1788201242.106564:0:14313:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b6c0 going to sleep for 11 seconds 00000080:00200000:3.0:1788201242.129944:0:14314:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201242.129984:0:14314:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201242.130009:0:14314:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:3.0:1788201242.130029:0:14314:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:3.0:1788201242.130039:0:14314:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201242.130049:0:14314:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201242.130054:0:14314:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bc4 00000080:00010000:3.0:1788201242.130064:0:14314:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bc4 from it ffffa0b181bf3b58 00010000:00010000:3.0:1788201242.130069:0:14314:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:3.0:1788201242.130079:0:14314:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:3.0:1788201242.130100:0:14314:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201242.130110:0:14314:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102b.replay-single, flags=25 00000080:00200000:3.0:1788201242.133220:0:14314:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448), inode mode 41ed mask 1 00000080:00200000:3.0:1788201242.133266:0:14314:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-3, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), flags=9 00000080:00200000:3.0:1788201242.133274:0:14314:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-3, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), intent=getattr 00000002:00010000:3.0:1788201242.133312:0:14314:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-3,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3f1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201242.136986:0:14314:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 14314 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.137013:0:14314:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 14314 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.137028:0:14314:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 14314 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201242.137065:0:14314:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14314:1875064442644352:192.168.201.130@tcp:101:chmod.0 00000100:00100000:3.0:1788201242.137169:0:14314:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6000 going to sleep for 11 seconds 00000100:00100000:2.0:1788201242.140305:0:14313:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6d80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14313:1875064442644096:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201242.140346:0:14313:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.140370:0:14313:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.140381:0:14313:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.140397:0:14313:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.140410:0:14313:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.140428:0:14313:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.140444:0:14313:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.140454:0:14313:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201242.140468:0:14313:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c6d80 x1875064442644096/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201258 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201242.140484:0:14313:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201242.140500:0:14313:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.140512:0:14313:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1011 (0 0) 00000080:00200000:2.0:1788201242.140520:0:14313:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201242.140635:0:14313:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3f3:0x0](ffff9023449a3248): rc = 0 00000080:00200000:2.0:1788201242.140657:0:14313:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3f3:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201242.140668:0:14313:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f3:0x0](ffff9023449a3248) for lock 0x4605c7c97e1c4c49 00000080:00010000:2.0:1788201242.140682:0:14313:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f3:0x0](ffff9023449a3248) 00000080:00010000:2.0:1788201242.140687:0:14313:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4c49 from it ffffa0b181bebb20 00010000:00010000:2.0:1788201242.140691:0:14313:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.140702:0:14313:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788201242.140753:0:14313:0:(file.c:6283:ll_getattr_dentry()) START file file-2:[0x200000bd1:0x3f3:0x0](ffff9023449a3248), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201242.140761:0:14313:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3f3:0x0](ffff9023449a3248),name=file-2 00000002:00010000:2.0:1788201242.140784:0:14313:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3f3:0x0]) in obj [0x200000bd1:0x3f1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201242.140795:0:14313:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201242.140808:0:14313:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201242.140815:0:14313:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201242.140819:0:14313:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f3:0x0](ffff9023449a3248) 00000080:00010000:2.0:1788201242.140823:0:14313:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f3:0x0](ffff9023449a3248) for lock 0x4605c7c97e1c4c49 00000080:00010000:2.0:1788201242.140827:0:14313:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4c49 from it ffffa0b181bebcb8 00010000:00010000:2.0:1788201242.140843:0:14313:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201242.140851:0:14313:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201242.140914:0:14313:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3f3:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201242.140925:0:14313:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3f3:0x0] 00000008:00010000:2.0:1788201242.140961:0:14313:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201242.140966:0:14313:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023533987b0, osc lock ffff9023500dcd10, flags 0x400001002 00010000:00010000:2.0:1788201242.141026:0:14313:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 3/1,0 mode: --/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.141038:0:14313:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 3/1,0 mode: --/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201242.141078:0:14313:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1788201242.141108:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442644480:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201242.150619:0:2361:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.150657:0:2361:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.150677:0:2361:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.150693:0:2361:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201242.150714:0:2361:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201242.150727:0:2361:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788201242.150740:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.150748:0:2361:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.150762:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.150772:0:2361:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201242.150797:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442644480:192.168.201.130@tcp:101:chmod.0 00000080:00200000:2.0:1788201242.150865:0:14313:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x3f3:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201242.150873:0:14313:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_release():245 00000020:00010000:2.0:1788201242.150896:0:14313:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201242.150903:0:14313:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023533987b0 (ffff902350b9d758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201242.150907:0:14313:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201242.150910:0:14313:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023533987b0 (ffff902350b9d638/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201242.150958:0:14313:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f3:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201242.151035:0:14313:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201242.151061:0:14313:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201242.151084:0:14313:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:2.0:1788201242.151098:0:14313:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:2.0:1788201242.151107:0:14313:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201242.151115:0:14313:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000100:00100000:3.0:1788201242.153731:0:14314:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14314:1875064442644352:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201242.153755:0:14314:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.153772:0:14314:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.153782:0:14314:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.153794:0:14314:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.153807:0:14314:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.153820:0:14314:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.153828:0:14314:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.153849:0:14314:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201242.153863:0:14314:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352688380 x1875064442644352/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201258 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201242.153892:0:14314:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201242.153908:0:14314:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.153924:0:14314:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1012 (0 0) 00000080:00200000:3.0:1788201242.153931:0:14314:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201242.154046:0:14314:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3f4:0x0](ffff9023449a1988): rc = 0 00000080:00200000:3.0:1788201242.154070:0:14314:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3f4:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201242.154084:0:14314:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f4:0x0](ffff9023449a1988) for lock 0x4605c7c97e1c4c57 00000080:00010000:3.0:1788201242.154097:0:14314:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f4:0x0](ffff9023449a1988) 00000080:00010000:3.0:1788201242.154101:0:14314:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4c57 from it ffffa0b181bf3b20 00010000:00010000:3.0:1788201242.154105:0:14314:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000080:00010000:2.0:1788201242.154748:0:14313:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bc4 00000080:00010000:2.0:1788201242.154756:0:14313:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bc4 from it ffffa0b181bebc08 00010000:00010000:2.0:1788201242.154761:0:14313:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:2.0:1788201242.154771:0:14313:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1788201242.154792:0:14313:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201242.154806:0:14313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102b.replay-single, flags=25 00000080:00200000:2.0:1788201242.154815:0:14313:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448), inode mode 41ed mask 1 00000080:00200000:2.0:1788201242.154820:0:14313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-2, flags=9 00000080:00200008:2.0:1788201242.154854:0:14313:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-2:[0x200000bd1:0x3f3:0x0](ffff9023449a3248) current size 0, valid attrs 41, mode 8180, uid 1204965376, gid -28637, new size 0, atime 1788201242.000000000, mtime 1788201242.000000000, ctime 1788201242.000000000 00010000:00010000:2.0:1788201242.154906:0:14313:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201242.154931:0:14313:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c4bf5 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3623 expref: -99 pid: 14296 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201242.154988:0:14313:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201242.154996:0:14313:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201242.154999:0:14313:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4c49 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3750 expref: -99 pid: 14313 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201242.155026:0:14313:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201242.155037:0:14313:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c6d80 x1875064442644864/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201242.155071:0:14313:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6d80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14313:1875064442644864:192.168.201.130@tcp:36:chmod.0 00000100:00100000:2.0:1788201242.155286:0:14313:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b6c0 going to sleep for 11 seconds 00010000:00010000:3.0:1788201242.157546:0:14314:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788201242.157614:0:14314:0:(file.c:6283:ll_getattr_dentry()) START file file-3:[0x200000bd1:0x3f4:0x0](ffff9023449a1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201242.157632:0:14314:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3f4:0x0](ffff9023449a1988),name=file-3 00000002:00010000:3.0:1788201242.157663:0:14314:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3f4:0x0]) in obj [0x200000bd1:0x3f1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201242.157683:0:14314:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201242.157697:0:14314:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201242.157706:0:14314:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201242.157714:0:14314:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f4:0x0](ffff9023449a1988) 00000080:00010000:3.0:1788201242.157718:0:14314:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f4:0x0](ffff9023449a1988) for lock 0x4605c7c97e1c4c57 00000080:00010000:3.0:1788201242.157727:0:14314:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4c57 from it ffffa0b181bf3cb8 00010000:00010000:3.0:1788201242.157733:0:14314:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201242.157743:0:14314:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201242.157805:0:14314:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3f4:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1788201242.157817:0:14314:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3f4:0x0] 00000008:00010000:3.0:1788201242.157873:0:14314:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788201242.157897:0:14314:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023483ef1b0, osc lock ffff9023488677c0, flags 0x400001002 00010000:00010000:3.0:1788201242.157962:0:14314:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 3/1,0 mode: --/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.157975:0:14314:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 3/1,0 mode: --/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788201242.158022:0:14314:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1788201242.158653:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442645248:192.168.201.130@tcp:101:chmod.0 00000080:00200000:2.0:1788201242.159081:0:14316:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201242.159101:0:14316:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201242.159116:0:14316:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1788201242.159129:0:14316:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1788201242.159136:0:14316:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201242.159143:0:14316:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201242.159147:0:14316:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bc4 00000080:00010000:2.0:1788201242.159154:0:14316:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bc4 from it ffffa0b181c0bb58 00010000:00010000:2.0:1788201242.159157:0:14316:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1788201242.159167:0:14316:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:2.0:1788201242.159182:0:14316:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201242.159191:0:14316:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102b.replay-single, flags=25 00000080:00200000:2.0:1788201242.159198:0:14316:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448), inode mode 41ed mask 1 00000080:00200000:2.0:1788201242.159211:0:14316:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-5, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), flags=9 00000080:00200000:2.0:1788201242.159216:0:14316:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-5, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), intent=getattr 00000002:00010000:2.0:1788201242.159231:0:14316:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-5,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3f1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201242.159278:0:14316:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 14316 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.159286:0:14316:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 14316 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.159297:0:14316:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 14316 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201242.159318:0:14316:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14316:1875064442645376:192.168.201.130@tcp:101:chmod.0 00000100:00100000:2.0:1788201242.169347:0:14316:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b240 going to sleep for 11 seconds 00010000:00010000:3.0:1788201242.175798:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.175850:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.175871:0:2362:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.175907:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1788201242.175935:0:2362:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788201242.175949:0:2362:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1788201242.175960:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.175970:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.175979:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.175990:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1788201242.176017:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442645248:192.168.201.130@tcp:101:chmod.0 00000080:00200000:3.0:1788201242.176084:0:14314:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x3f4:0x0] updating i_size 0 i_blocks 1 00000020:00010000:3.0:1788201242.176090:0:14314:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235000e9a0 (ffff902346343248/0) at cl_lock_release():245 00000020:00010000:3.0:1788201242.176094:0:14314:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235000e9a0 (ffff902346343248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201242.176100:0:14314:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023483ef1b0 (ffff902346343e18/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788201242.176105:0:14314:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235000e9a0 (ffff902346343248/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201242.176109:0:14314:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023483ef1b0 (ffff902346343248/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201242.179492:0:14317:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201242.179511:0:14317:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201242.179528:0:14317:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1788201242.179543:0:14317:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:2.0:1788201242.179552:0:14317:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201242.179562:0:14317:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201242.179567:0:14317:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bc4 00000080:00010000:2.0:1788201242.179575:0:14317:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bc4 from it ffffa0b181c13b58 00010000:00010000:2.0:1788201242.179579:0:14317:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1788201242.179589:0:14317:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:2.0:1788201242.179612:0:14317:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201242.179622:0:14317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102b.replay-single, flags=25 00000080:00200000:2.0:1788201242.179629:0:14317:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448), inode mode 41ed mask 1 00000080:00200000:2.0:1788201242.179645:0:14317:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-6, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), flags=9 00000080:00200000:2.0:1788201242.179651:0:14317:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-6, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), intent=getattr 00000080:00200000:3.0:1788201242.179846:0:14314:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f4:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201242.180488:0:14314:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201242.180506:0:14314:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201242.180518:0:14314:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:3.0:1788201242.180641:0:14314:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:3.0:1788201242.180650:0:14314:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201242.180656:0:14314:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201242.180659:0:14314:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bc4 00000080:00010000:3.0:1788201242.180664:0:14314:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bc4 from it ffffa0b181bf3c08 00010000:00010000:3.0:1788201242.180668:0:14314:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:3.0:1788201242.180677:0:14314:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:3.0:1788201242.180689:0:14314:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201242.180695:0:14314:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102b.replay-single, flags=25 00000080:00200000:3.0:1788201242.180700:0:14314:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448), inode mode 41ed mask 1 00000080:00200000:3.0:1788201242.180703:0:14314:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-3, flags=9 00000100:00100000:2.0:1788201242.182480:0:14316:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14316:1875064442645376:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201242.182508:0:14316:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.182528:0:14316:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.182538:0:14316:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.182552:0:14316:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.182564:0:14316:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.182577:0:14316:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.182586:0:14316:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.182599:0:14316:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201242.182613:0:14316:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c6a00 x1875064442645376/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201258 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201242.182627:0:14316:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201242.182643:0:14316:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.182654:0:14316:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1014 (0 0) 00000080:00200000:2.0:1788201242.182662:0:14316:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201242.182769:0:14316:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3f6:0x0](ffff9023449a42c8): rc = 0 00000080:00200000:2.0:1788201242.182794:0:14316:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3f6:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201242.182805:0:14316:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f6:0x0](ffff9023449a42c8) for lock 0x4605c7c97e1c4c73 00000080:00010000:2.0:1788201242.182819:0:14316:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f6:0x0](ffff9023449a42c8) 00000080:00010000:2.0:1788201242.182823:0:14316:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4c73 from it ffffa0b181c0bb20 00010000:00010000:2.0:1788201242.182827:0:14316:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.185733:0:14316:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788201242.186054:0:14314:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-3:[0x200000bd1:0x3f4:0x0](ffff9023449a1988) current size 0, valid attrs 41, mode 8180, uid 1094316032, gid -28637, new size 0, atime 1788201242.000000000, mtime 1788201242.000000000, ctime 1788201242.000000000 00010000:00010000:3.0:1788201242.186091:0:14314:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201242.186129:0:14314:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c4c03 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3654 expref: -99 pid: 14297 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201242.186183:0:14314:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201242.186190:0:14314:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201242.186194:0:14314:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4c57 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3773 expref: -99 pid: 14314 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201242.186217:0:14314:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 3 is allocated opc 36, max 7 00000100:00100000:3.0:1788201242.186229:0:14314:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352688380 x1875064442645888/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201242.186259:0:14314:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14314:1875064442645888:192.168.201.130@tcp:36:chmod.0 00000100:00100000:3.0:1788201242.186357:0:14314:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6540 going to sleep for 11 seconds 00010000:00010000:3.0:1788201242.186408:0:14320:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201242.186420:0:14320:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201242.186429:0:14320:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4c73 from it ffffa0b181d0bd78 00010000:00010000:3.0:1788201242.186434:0:14320:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201242.186442:0:14320:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201242.186500:0:14320:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 12 type: IBT flags: 0x10000000000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.186510:0:14320:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x0 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.186535:0:14320:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4cb9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 13 type: IBT flags: 0x10000000000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.186542:0:14320:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4cb9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x0 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.186558:0:14320:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4cc0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 14 type: IBT flags: 0x10000000000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.186566:0:14320:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4cc0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x0 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.186581:0:14320:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4cc7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 15 type: IBT flags: 0x10000000000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.186588:0:14320:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4cc7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT flags: 0x0 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.186606:0:14320:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4cce lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 16 type: IBT flags: 0x10000000000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.186613:0:14320:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4cce lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x0 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.186629:0:14320:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c4cd5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 17 type: IBT flags: 0x10000000000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.186636:0:14320:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c4cd5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT flags: 0x0 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.186656:0:14320:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c4cdc lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 18 type: IBT flags: 0x10000000000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.186663:0:14320:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c4cdc lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT flags: 0x0 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201242.186717:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201242.186733:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442646016:192.168.201.130@tcp:63:ll_sa.0 00000080:00200000:3.0:1788201242.192550:0:14320:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3f6:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1788201242.192574:0:14320:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3f6:0x0] 00000008:00010000:3.0:1788201242.192617:0:14320:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:3.0:1788201242.192622:0:14320:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023483efa30, osc lock ffff9023488677c0, flags 0x400041012 00010000:00010000:3.0:1788201242.192711:0:14320:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000750e72d7/0x4605c7c97e1c4d1b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 14320 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.192728:0:14320:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000750e72d7/0x4605c7c97e1c4d1b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 14320 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:3.0:1788201242.192770:0:14320:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235000e9a0 (ffff902346343248/0) at cl_lock_release():245 00000020:00010000:3.0:1788201242.192775:0:14320:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235000e9a0 (ffff902346343248/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201242.192778:0:14320:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023483efa30 (ffff902346343e18/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788201242.192783:0:14320:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235000e9a0 (ffff902346343248/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201242.192786:0:14320:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023483efa30 (ffff902346343248/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201242.192822:0:14320:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f6:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1788201242.192899:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442646528:192.168.201.130@tcp:101:ll_sa.0 00000080:00200008:3.0:1788201242.193131:0:14316:0:(file.c:6283:ll_getattr_dentry()) START file file-5:[0x200000bd1:0x3f6:0x0](ffff9023449a42c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201242.193142:0:14316:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3f6:0x0](ffff9023449a42c8),name=file-5 00000002:00010000:3.0:1788201242.193170:0:14316:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3f6:0x0]) in obj [0x200000bd1:0x3f1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201242.193189:0:14316:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201242.193201:0:14316:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201242.193209:0:14316:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201242.193218:0:14316:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f6:0x0](ffff9023449a42c8) 00000080:00010000:3.0:1788201242.193223:0:14316:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f6:0x0](ffff9023449a42c8) for lock 0x4605c7c97e1c4c73 00000080:00010000:3.0:1788201242.193231:0:14316:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4c73 from it ffffa0b181c0bcb8 00010000:00010000:3.0:1788201242.193248:0:14316:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201242.193258:0:14316:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788201242.193296:0:14316:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3f6:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1788201242.193302:0:14316:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3f6:0x0] 00000008:00010000:3.0:1788201242.193320:0:14316:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788201242.193322:0:14316:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023483efa30, osc lock ffff9023488677c0, flags 0x400001002 00010000:00010000:3.0:1788201242.193331:0:14316:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902351d6bc00 type 11 mode 6 res 10737419264/1051 (0 18446744073709551615) 00010000:00010000:3.0:1788201242.193356:0:14316:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c128faeb/0x4605c7c97e1c4d22 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 14316 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.193366:0:14316:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c128faeb/0x4605c7c97e1c4d22 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 14316 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788201242.193395:0:14316:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1788201242.193407:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201242.193417:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442646656:192.168.201.130@tcp:101:chmod.0 00000002:00010000:2.0:1788201242.199376:0:14317:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-6,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3f1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201242.199459:0:14317:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 23 type: IBT flags: 0x10000000000000 pid: 14317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.199471:0:14317:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 23 type: IBT flags: 0x0 pid: 14317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.199484:0:14317:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 23 type: IBT flags: 0x0 pid: 14317 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201242.199507:0:14317:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14317:1875064442647040:192.168.201.130@tcp:101:chmod.0 00000100:00100000:2.0:1788201242.199625:0:14317:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b780 going to sleep for 11 seconds 00010000:00010000:2.0:1788201242.199689:0:14324:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c4c7a lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d37e3 expref: -99 pid: 14318 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201242.199703:0:14324:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c4c7a lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d37e3 expref: -99 pid: 14318 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00010000:2.0:1788201242.199713:0:14324:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4c7a from it ffffa0b181c43d78 00010000:00010000:2.0:1788201242.199719:0:14324:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c4c7a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d37e3 expref: -99 pid: 14318 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201242.199728:0:14324:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c4c7a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d37e3 expref: -99 pid: 14318 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201242.199775:0:14324:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4d30 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 24 type: IBT flags: 0x10000000000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.199782:0:14324:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4d30 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x0 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.199799:0:14324:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4d37 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 25 type: IBT flags: 0x10000000000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.199806:0:14324:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4d37 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x0 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.199823:0:14324:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4d3e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 26 type: IBT flags: 0x10000000000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.199840:0:14324:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4d3e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 26 type: IBT flags: 0x0 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.199859:0:14324:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c4d45 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 27 type: IBT flags: 0x10000000000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.199866:0:14324:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c4d45 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 27 type: IBT flags: 0x0 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.199909:0:14324:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4d4c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 28 type: IBT flags: 0x10000000000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.199917:0:14324:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4d4c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 28 type: IBT flags: 0x0 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.199935:0:14324:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4d53 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 29 type: IBT flags: 0x10000000000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.199942:0:14324:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4d53 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 29 type: IBT flags: 0x0 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.199960:0:14324:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4d5a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 30 type: IBT flags: 0x10000000000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.199967:0:14324:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4d5a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 30 type: IBT flags: 0x0 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201242.200022:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201242.200038:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442647168:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:2.0:1788201242.208339:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000750e72d7/0x4605c7c97e1c4d1b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 14320 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.208352:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000750e72d7/0x4605c7c97e1c4d1b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d37ff expref: -99 pid: 14320 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.208360:0:2362:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000750e72d7/0x4605c7c97e1c4d1b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d37ff expref: -99 pid: 14320 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.208367:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000750e72d7/0x4605c7c97e1c4d1b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d37ff expref: -99 pid: 14320 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201242.208408:0:2362:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000750e72d7/0x4605c7c97e1c4d1b lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d37ff expref: -99 pid: 14320 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.208443:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000750e72d7/0x4605c7c97e1c4d1b lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d37ff expref: -99 pid: 14320 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.208451:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000750e72d7/0x4605c7c97e1c4d1b lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d37ff expref: -99 pid: 14320 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.208471:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000750e72d7/0x4605c7c97e1c4d1b lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d37ff expref: -99 pid: 14320 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.208493:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000750e72d7/0x4605c7c97e1c4d1b lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d37ff expref: -99 pid: 14320 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201242.208518:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442646528:192.168.201.130@tcp:101:ll_sa.0 00010000:00010000:2.0:1788201242.208536:0:2362:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d381b expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.208555:0:2362:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d381b expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.208563:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d381b expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.208569:0:2362:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d381b expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.208575:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d381b expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.213062:0:14320:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e000/0x4605c7c97e1c4d68 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 18 type: IBT flags: 0x10000000000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.213081:0:14320:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e000/0x4605c7c97e1c4d68 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT flags: 0x0 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201242.213178:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442647680:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:2.0:1788201242.213332:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d381b expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.213355:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d381b expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.213371:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d381b expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201242.213393:0:2362:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352688700 x1875064442646016/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3016/7008 e 0 to 0 dl 1788201258 ref 1 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:2.0:1788201242.213407:0:2362:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d381b expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201242.213471:0:14317:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14317:1875064442647040:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201242.213496:0:14317:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 18 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.213510:0:14317:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.213519:0:14317:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.213532:0:14317:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.213558:0:14317:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.213569:0:14317:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.213577:0:14317:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.213585:0:14317:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201242.213594:0:14317:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c6a00 x1875064442647040/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201258 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201242.213604:0:14317:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201242.213619:0:14317:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.213634:0:14317:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c4cf8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d387d expref: -99 pid: 14325 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201242.213647:0:14317:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c4cf8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d387d expref: -99 pid: 14325 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201242.213656:0:14317:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.213664:0:14317:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.213671:0:14317:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.213692:0:14317:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1788201242.213701:0:14317:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201242.214035:0:14317:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3f7:0x0](ffff9023449a3a88): rc = 0 00000080:00200000:2.0:1788201242.214060:0:14317:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3f7:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201242.214072:0:14317:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f7:0x0](ffff9023449a3a88) for lock 0x4605c7c97e1c4cf8 00000080:00010000:2.0:1788201242.214087:0:14317:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f7:0x0](ffff9023449a3a88) 00000080:00010000:2.0:1788201242.214092:0:14317:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4cf8 from it ffffa0b181c13b20 00010000:00010000:2.0:1788201242.214096:0:14317:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c4cf8 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d387d expref: -99 pid: 14325 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201242.214105:0:14317:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c4cf8 lrc: 3/0,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d387d expref: -99 pid: 14325 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200008:2.0:1788201242.214164:0:14317:0:(file.c:6283:ll_getattr_dentry()) START file file-6:[0x200000bd1:0x3f7:0x0](ffff9023449a3a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201242.214173:0:14317:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3f7:0x0](ffff9023449a3a88),name=file-6 00000002:00010000:2.0:1788201242.214194:0:14317:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3f7:0x0]) in obj [0x200000bd1:0x3f1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201242.214204:0:14317:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c4cf8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d387d expref: -99 pid: 14325 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201242.214214:0:14317:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c4cf8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d387d expref: -99 pid: 14325 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201242.214221:0:14317:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201242.214227:0:14317:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f7:0x0](ffff9023449a3a88) 00000080:00010000:2.0:1788201242.214231:0:14317:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f7:0x0](ffff9023449a3a88) for lock 0x4605c7c97e1c4cf8 00000080:00010000:2.0:1788201242.214235:0:14317:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4cf8 from it ffffa0b181c13cb8 00010000:00010000:2.0:1788201242.214239:0:14317:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c4cf8 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d387d expref: -99 pid: 14325 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201242.214247:0:14317:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c4cf8 lrc: 3/0,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d387d expref: -99 pid: 14325 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201242.214275:0:14317:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3f7:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201242.214288:0:14317:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3f7:0x0] 00000008:00010000:2.0:1788201242.214323:0:14317:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201242.214328:0:14317:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398b30, osc lock ffff9023500dcd10, flags 0x400001002 00010000:00010000:2.0:1788201242.214388:0:14317:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c4d6f lrc: 3/1,0 mode: --/PR res: [0x240000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 14317 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.214399:0:14317:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c4d6f lrc: 3/1,0 mode: --/PR res: [0x240000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 14317 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201242.214424:0:14317:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788201242.214448:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201242.214455:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.214466:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351685600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201242.214471:0:5599:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.214483:0:5599:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.214492:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 3/0,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.214511:0:5599:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c4cf8 lrc: 3/0,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d387d expref: -99 pid: 14325 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201242.214542:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 2/0,0 mode: --/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.214550:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201242.214560:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.214569:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4d29 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d38ca expref: -99 pid: 14317 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.214587:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201242.214589:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201242.214591:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00010000:2.0:1788201242.214606:0:2362:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4cb2 from it ffff902342dfa990 00010000:00010000:2.0:1788201242.214611:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d381b expref: -99 pid: 14320 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.214618:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d381b expref: -99 pid: 14320 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201242.214646:0:2362:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4cb9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3837 expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.214658:0:2362:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4cb9 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3837 expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.214667:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4cb9 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3837 expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.214675:0:2362:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4cb9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3837 expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.214684:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4cb9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3837 expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.214691:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4cb9 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3837 expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.214698:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4cb9 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3837 expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.214705:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4cb9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3837 expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201242.215241:0:2362:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352688700 x1875064442646016/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3016/7008 e 0 to 0 dl 1788201258 ref 2 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:2.0:1788201242.215255:0:2362:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4cb9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3837 expref: -99 pid: 14320 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215285:0:11305:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d381b expref: -99 pid: 14320 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201242.215370:0:11305:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3f7:0x0](ffff9023449a3a88): rc = 0 00000080:00200000:2.0:1788201242.215386:0:11305:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3f7:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201242.215395:0:11305:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f7:0x0](ffff9023449a3a88) for lock 0x4605c7c97e1c4cb2 00000080:00010000:2.0:1788201242.215402:0:11305:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4cb2 from it ffff902342dfa990 00010000:00010000:2.0:1788201242.215405:0:11305:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d381b expref: -99 pid: 14320 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201242.215413:0:11305:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d381b expref: -99 pid: 14320 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1788201242.215440:0:2362:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4cb9 from it ffff902342df9190 00010000:00010000:2.0:1788201242.215444:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4cb9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3837 expref: -99 pid: 14320 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215451:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4cb9 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3837 expref: -99 pid: 14320 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201242.215468:0:2362:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4cc0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x0 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215474:0:2362:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4cc0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215480:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4cc0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x10206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215486:0:2362:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4cc0 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x10206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215494:0:2362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4cc0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215500:0:2362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db0a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201242.215503:0:2362:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4cc0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 16 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215510:0:2362:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4cc0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 16 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215517:0:2362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4cc0 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 16 type: IBT flags: 0x20e400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215523:0:2362:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201242.215528:0:2362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4cc0 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 16 type: IBT flags: 0x4a0f400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215533:0:2362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c4cc0 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 16 type: IBT flags: 0x4a0f400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215546:0:2362:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4cc7 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT flags: 0x0 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215551:0:2362:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4cc7 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215557:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4cc7 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT flags: 0x10206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215562:0:2362:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4cc7 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT flags: 0x10206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215568:0:2362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4cc7 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215573:0:2362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db0c00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201242.215576:0:2362:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4cc7 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 15 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215581:0:2362:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4cc7 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 15 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215587:0:2362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4cc7 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 15 type: IBT flags: 0x20e400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215592:0:2362:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201242.215594:0:2362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4cc7 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 15 type: IBT flags: 0x4a0f400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215600:0:2362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4cc7 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 15 type: IBT flags: 0x4a0f400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215610:0:2362:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4cce lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x0 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215615:0:2362:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4cce lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215620:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4cce lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x10206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215625:0:2362:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4cce lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x10206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215631:0:2362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4cce lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215636:0:2362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db0400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201242.215638:0:2362:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4cce lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 14 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215644:0:2362:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4cce lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 14 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215649:0:2362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4cce lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 14 type: IBT flags: 0x20e400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215654:0:2362:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201242.215656:0:2362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4cce lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 14 type: IBT flags: 0x4a0f400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215662:0:2362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c4cce lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 14 type: IBT flags: 0x4a0f400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215673:0:2362:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c4cd5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x0 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215679:0:2362:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c4cd5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215684:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c4cd5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x10206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215690:0:2362:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c4cd5 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x10206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215696:0:2362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c4cd5 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215701:0:2362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db1200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201242.215704:0:2362:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c4cd5 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.215709:0:2362:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c4cd5 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.216346:0:2362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c4cd5 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x20e400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.216354:0:2362:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201242.216360:0:2362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c4cd5 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x4a0f400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.216367:0:2362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c4cd5 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x4a0f400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.216390:0:2362:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c4cdc lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x0 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.216397:0:2362:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c4cdc lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.216403:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c4cdc lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x10206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.216409:0:2362:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c4cdc lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x10206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.216417:0:2362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c4cdc lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.216423:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4d30 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x0 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.216425:0:2362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db1800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201242.216432:0:2362:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c4cdc lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.216462:0:2362:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c4cdc lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.216470:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4d30 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.216473:0:2362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c4cdc lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x20e400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.216492:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4d30 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x10206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.216502:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4d30 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x10206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.216511:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4d30 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.217560:0:2362:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201242.217573:0:2362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c4cdc lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 12 type: IBT flags: 0x4a0f400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.217584:0:2362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c4cdc lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 12 type: IBT flags: 0x4a0f400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201242.217615:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442646016:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:2.0:1788201242.217676:0:11305:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4cb9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3837 expref: -99 pid: 14320 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201242.217867:0:11305:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3f8:0x0](ffff9023449b4b08): rc = 0 00000080:00200000:2.0:1788201242.217984:0:11305:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3f8:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201242.217996:0:11305:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f8:0x0](ffff9023449b4b08) for lock 0x4605c7c97e1c4cb9 00000080:00010000:2.0:1788201242.218004:0:11305:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4cb9 from it ffff902342df9190 00010000:00010000:2.0:1788201242.218008:0:11305:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4cb9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3837 expref: -99 pid: 14320 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201242.218019:0:11305:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4cb9 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3837 expref: -99 pid: 14320 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1788201242.218062:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442647808:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201242.218074:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351684c00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201242.218087:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4d30 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218103:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4d30 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218113:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4d30 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x20e400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218120:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201242.218130:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4d30 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x4a0f400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218139:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4d30 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x4a0f400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218167:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4d37 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x0 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218174:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4d37 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218180:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4d37 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x10206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218186:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4d37 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x10206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218193:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4d37 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.218198:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c128faeb/0x4605c7c97e1c4d22 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 14316 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218199:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351685a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201242.218206:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4d37 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.218237:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c128faeb/0x4605c7c97e1c4d22 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3899 expref: -99 pid: 14316 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.218255:0:2362:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c128faeb/0x4605c7c97e1c4d22 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3899 expref: -99 pid: 14316 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.218269:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c128faeb/0x4605c7c97e1c4d22 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3899 expref: -99 pid: 14316 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201242.218287:0:2362:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c128faeb/0x4605c7c97e1c4d22 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3899 expref: -99 pid: 14316 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201242.218300:0:2362:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788201242.218315:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c128faeb/0x4605c7c97e1c4d22 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3899 expref: -99 pid: 14316 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.218323:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c128faeb/0x4605c7c97e1c4d22 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3899 expref: -99 pid: 14316 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.218331:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c128faeb/0x4605c7c97e1c4d22 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3899 expref: -99 pid: 14316 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.218339:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c128faeb/0x4605c7c97e1c4d22 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3899 expref: -99 pid: 14316 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201242.218350:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442646656:192.168.201.130@tcp:101:chmod.0 00000100:00100000:2.0:1788201242.218365:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201242.218372:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442647936:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201242.218422:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4d37 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218430:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4d37 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x20e400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218436:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201242.218441:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4d37 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x4a0f400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218446:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4d37 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x4a0f400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218462:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4d3e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x0 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218467:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4d3e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218473:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4d3e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x10206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218479:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4d3e lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x10206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218485:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4d3e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218490:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351685400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201242.218494:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4d3e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218500:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4d3e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218505:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4d3e lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x20e400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218511:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201242.218514:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4d3e lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x4a0f400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218519:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4d3e lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x4a0f400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218530:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c4d45 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x0 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218537:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c4d45 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218542:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c4d45 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x10206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218547:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c4d45 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x10206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218552:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c4d45 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218556:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351685000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201242.218559:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c4d45 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218565:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c4d45 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218570:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c4d45 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x20e400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218576:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201242.218579:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c4d45 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x4a0f400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218585:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c4d45 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x4a0f400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218595:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4d4c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x0 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218600:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4d4c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218605:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4d4c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x10206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218610:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4d4c lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x10206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218615:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4d4c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218619:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351684400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201242.218622:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4d4c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218627:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4d4c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218631:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4d4c lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x20e400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218636:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201242.218639:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4d4c lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x4a0f400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218646:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c4d4c lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x4a0f400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218655:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4d53 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218660:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4d53 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218665:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4d53 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218669:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4d53 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218673:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4d53 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218677:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351685e00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201242.218679:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4d53 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218684:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4d53 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218689:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4d53 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x20e400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218695:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201242.218698:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4d53 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.218704:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c4d53 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.222260:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4d5a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.222309:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4d5a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.222317:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4d5a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.222325:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4d5a lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.222334:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4d5a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.222342:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351684a00 already unused, calling callback (ffffffffc1a662a0) 00000100:00100000:2.0:1788201242.222350:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00010000:00010000:3.0:1788201242.222363:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4d5a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.222375:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4d5a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.222386:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4d5a lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x20e400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201242.222393:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458221056 00010000:00010000:3.0:1788201242.222395:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000100:00100000:2.0:1788201242.222403:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458221056 00010000:00010000:3.0:1788201242.222406:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4d5a lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.222416:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4d5a lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201242.222460:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1318 00000100:00100000:3.0:1788201242.222465:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442647168:192.168.201.130@tcp:63:ll_sa.0 00000100:00100000:2.0:1788201242.222476:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c7100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:35604eb4-baf9-45be-a20a-4883a2cd8cf9+5:12968:x1875064458221056:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201242.222495:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c4cb9 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x9383f985bc4d3837 expref: -99 pid: 14320 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201242.222575:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201242.222581:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c7100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:35604eb4-baf9-45be-a20a-4883a2cd8cf9+5:12968:x1875064458221056:12345-192.168.201.130@tcp:104: Request processed in 109us (238us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201242.222588:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1318 00010000:00010000:2.0:1788201242.222638:0:14316:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201242.222651:0:14316:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201242.222655:0:14316:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4c73 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d37c7 expref: -99 pid: 14316 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:2.0:1788201242.222683:0:14316:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 6 is allocated opc 36, max 7 00000100:00100000:2.0:1788201242.222694:0:14316:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c5c00 x1875064442648192/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201242.222719:0:14316:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5c00 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14316:1875064442648192:192.168.201.130@tcp:36:chmod.0 00000100:00100000:2.0:1788201242.222773:0:14316:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bd80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201242.232783:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442647936:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201242.232791:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e000/0x4605c7c97e1c4d68 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.232810:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e000/0x4605c7c97e1c4d68 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.232817:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e000/0x4605c7c97e1c4d68 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.232826:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e000/0x4605c7c97e1c4d68 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.232846:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e000/0x4605c7c97e1c4d68 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.232850:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c4d6f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 14317 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.232854:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351f9e000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201242.232863:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e000/0x4605c7c97e1c4d68 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.232910:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e000/0x4605c7c97e1c4d68 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.232919:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e000/0x4605c7c97e1c4d68 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x20e400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.232926:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201242.232932:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e000/0x4605c7c97e1c4d68 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201242.232938:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e000/0x4605c7c97e1c4d68 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 14320 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201242.233037:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442647680:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:2.0:1788201242.233172:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c4d6f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3902 expref: -99 pid: 14317 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.233193:0:2362:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c4d6f lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3902 expref: -99 pid: 14317 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.233209:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c4d6f lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3902 expref: -99 pid: 14317 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201242.233228:0:2362:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c4d6f lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3902 expref: -99 pid: 14317 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201242.233248:0:2362:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788201242.233259:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c4d6f lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3902 expref: -99 pid: 14317 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.233270:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c4d6f lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3902 expref: -99 pid: 14317 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.233278:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c4d6f lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3902 expref: -99 pid: 14317 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201242.233287:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c4d6f lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3902 expref: -99 pid: 14317 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201242.233307:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442647808:192.168.201.130@tcp:101:chmod.0 00000080:00200000:2.0:1788201242.233368:0:14317:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x3f7:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201242.233374:0:14317:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_release():245 00000020:00010000:2.0:1788201242.233378:0:14317:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201242.233385:0:14317:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398b30 (ffff902350b9d758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201242.233389:0:14317:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201242.233392:0:14317:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398b30 (ffff902350b9d638/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201242.233439:0:14317:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f7:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201242.233862:0:14317:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201242.234147:0:14317:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201242.234171:0:14317:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:2.0:1788201242.234188:0:14317:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:2.0:1788201242.234197:0:14317:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201242.234205:0:14317:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201242.234209:0:14317:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bc4 00000080:00010000:2.0:1788201242.234216:0:14317:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bc4 from it ffffa0b181c13c08 00010000:00010000:2.0:1788201242.234220:0:14317:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:2.0:1788201242.234227:0:14317:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:2.0:1788201242.234242:0:14317:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201242.234251:0:14317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102b.replay-single, flags=25 00000080:00200000:2.0:1788201242.234258:0:14317:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448), inode mode 41ed mask 1 00000080:00200000:2.0:1788201242.234262:0:14317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-6, flags=9 00000080:00200008:2.0:1788201242.234279:0:14317:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-6:[0x200000bd1:0x3f7:0x0](ffff9023449a3a88) current size 0, valid attrs 41, mode 8180, uid 1204969472, gid -28637, new size 0, atime 1788201242.000000000, mtime 1788201242.000000000, ctime 1788201242.000000000 00010000:00010000:2.0:1788201242.234304:0:14317:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d381b expref: -99 pid: 14320 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201242.234323:0:14317:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c4c2d lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d36e7 expref: -99 pid: 14300 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201242.234342:0:14317:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c4cf8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d387d expref: -99 pid: 14325 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201242.234352:0:14317:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c4c2d lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d36e7 expref: -99 pid: 14300 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201242.234395:0:14317:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d381b expref: -99 pid: 14320 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201242.234403:0:14317:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c4cf8 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d387d expref: -99 pid: 14325 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201242.234408:0:14317:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201242.234410:0:14317:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c4cb2 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d381b expref: -99 pid: 14320 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201242.234425:0:14317:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c4cf8 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d387d expref: -99 pid: 14325 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000020:00100000:2.0:1788201242.234444:0:14317:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 7 is allocated opc 36, max 7 00000100:00100000:2.0:1788201242.234455:0:14317:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c6a00 x1875064442648576/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201242.234484:0:14317:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14317:1875064442648576:192.168.201.130@tcp:36:chmod.0 00000100:00100000:2.0:1788201242.234724:0:14317:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b180 going to sleep for 11 seconds 00010000:00010000:3.0:1788201244.097261:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201245.121394:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201248.195117:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201252.289097:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201253.249038:0:14317:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b180 going to sleep for 1 seconds 00000100:00100000:2.0:1788201253.249523:0:14312:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342e369c0 going to sleep for 1 seconds 00000100:00100000:3.0:1788201253.251275:0:14316:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bd80 going to sleep for 1 seconds 00000100:00100000:2.0:1788201254.273118:0:14312:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342e369c0 going to sleep for 1 seconds 00000100:00100000:2.0:1788201254.274248:0:14315:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234502fb40 going to sleep for 1 seconds 00000100:00100000:2.0:1788201254.279133:0:14316:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bd80 going to sleep for 1 seconds 00000100:00100000:2.0:1788201255.297147:0:14316:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bd80 going to sleep for 1 seconds 00000100:00100000:2.0:1788201255.300731:0:14318:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234502f480 going to sleep for 1 seconds 00000100:00100000:2.0:1788201255.309646:0:14313:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b6c0 going to sleep for 1 seconds 00000100:00100000:2.0:1788201256.321269:0:14313:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b6c0 going to sleep for 1 seconds 00000100:00100000:2.0:1788201256.321620:0:14317:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b180 going to sleep for 1 seconds 00000100:00100000:3.0:1788201256.321673:0:14318:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234502f480 going to sleep for 1 seconds 00000100:00100000:2.0:1788201256.321947:0:14312:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342e369c0 going to sleep for 1 seconds 00010000:00010000:3.0:1788201256.385222:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201257.348224:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442649856:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201257.348355:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442650112:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201257.348390:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788201257.348396:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442649728:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201257.348424:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442649984:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201257.348862:0:14312:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342e369c0 going to sleep for 1 seconds 00000100:00100000:2.0:1788201257.348895:0:14317:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b180 going to sleep for 1 seconds 00000100:00100000:2.0:1788201257.348907:0:14313:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b6c0 going to sleep for 1 seconds 00000100:00100000:2.0:1788201257.348928:0:14314:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6540 going to sleep for 1 seconds 00000100:00100000:3.0:1788201257.351250:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442649728:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201257.351657:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442649984:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201257.351732:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442650112:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201257.352051:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442649856:192.168.201.130@tcp:400:kworker.0 00000100:00000400:2.0:1788201258.369920:0:14313:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201242/real 1788201242] req@ffff9023458c6d80 x1875064442644864/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201258 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1788201258.373969:0:14314:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201242/real 1788201242] req@ffff902352688380 x1875064442645888/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201258 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201258.441460:0:14313:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 3, was 3): DISCONN 00000100:00000400:2.0:1788201258.442065:0:14317:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201242/real 1788201242] req@ffff9023458c6a00 x1875064442648576/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201258 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201258.442096:0:14317:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 3, was 3): DISCONN 00000100:00100000:2.0:1788201258.442101:0:14317:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b180 going to sleep for 0 seconds 00000100:00000400:2.0:1788201258.442141:0:14312:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201242/real 1788201242] req@ffff9023488b3480 x1875064442645120/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201258 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201258.442157:0:14312:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 3, was 3): DISCONN 00000100:00100000:2.0:1788201258.442160:0:14312:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342e369c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788201258.442229:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201258.442234:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201258.442237:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201258.442243:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 508 00000100:00080000:2.0:1788201258.442255:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788201258.442268:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201258.442285:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023458c7b80 x1875064442650368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201258.442333:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442650240:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201258.442417:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442650624:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201258.442450:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201258.442455:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442650496:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201258.442484:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442650368:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201258.443177:0:14313:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788201258.443766:0:14314:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 4, was 3): CONNECTING 00010000:00010000:3.0:1788201258.443830:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201258.443838:0:14314:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6540 going to sleep for 0 seconds 00000100:00100000:3.0:1788201258.443891:0:14316:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bd80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201258.460387:0:14318:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234502f480 going to sleep for 0 seconds 00000100:00100000:3.0:1788201258.462668:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442650496:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201258.462699:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442650240:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201258.462710:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442650624:192.168.201.130@tcp:400:kworker.0 00000100:00080000:2.0:1788201258.483026:0:2358:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff902350a11000: connect to target with instance 2 00000100:00080000:2.0:1788201258.483060:0:2358:0:(import.c:1297:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@192.168.201.130@tcp after partition 00000100:00080000:2.0:1788201258.483064:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to RECOVER 00000100:00080000:2.0:1788201258.483070:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:00080000:3.0:1788201258.500822:0:14313:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442644864 new x1875064442650752 00000100:00100000:3.0:1788201258.500946:0:14313:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b6c0 going to sleep for 11 seconds 00000100:00080000:3.0:1788201258.501094:0:14314:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442645888 new x1875064442650880 00000100:00100000:3.0:1788201258.501146:0:14314:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6540 going to sleep for 11 seconds 00000100:00080000:3.0:1788201258.501161:0:14318:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442647552 new x1875064442651008 00000100:00100000:3.0:1788201258.505739:0:14318:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234502f480 going to sleep for 11 seconds 00000100:00080000:3.0:1788201258.505818:0:14316:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442648192 new x1875064442651136 00000100:00100000:3.0:1788201258.505960:0:14316:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bd80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201258.517426:0:14314:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14314:1875064442645888:192.168.201.130@tcp:36:chmod.0 00000080:00200000:3.0:1788201258.517513:0:14314:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3f4:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201258.517530:0:14314:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201258.517624:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201258.517650:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442651264:192.168.201.130@tcp:2:chmod.0 00000100:00100000:3.0:1788201258.518108:0:14318:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345833b80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14318:1875064442647552:192.168.201.130@tcp:36:chmod.0 00000100:00100000:3.0:1788201258.518190:0:14316:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5c00 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14316:1875064442648192:192.168.201.130@tcp:36:chmod.0 00000080:00200000:3.0:1788201258.518235:0:14316:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3f6:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201258.518243:0:14316:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201258.518298:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442651392:192.168.201.130@tcp:2:chmod.0 00000080:00200000:3.0:1788201258.518431:0:14318:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3f8:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201258.518439:0:14318:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201258.518484:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201258.518492:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442651520:192.168.201.130@tcp:2:chmod.0 00000100:00080000:2.0:1788201258.524355:0:14317:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442648576 new x1875064442651904 00000100:00100000:2.0:1788201258.524517:0:14317:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b180 going to sleep for 11 seconds 00000100:00100000:2.0:1788201258.536795:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442651264:192.168.201.130@tcp:2:chmod.0 00000080:00200000:3.0:1788201258.537111:0:14314:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f4:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201258.537132:0:14314:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-3:[0x200000bd1:0x3f4:0x0](ffff9023449a1988) current size 0, valid attrs 10000041, mode 8180, uid 1094316032, gid -28637, new size 0, atime 1788201242.000000000, mtime 1788201242.000000000, ctime 1788201242.000000000, rc 0 00000100:00100000:3.0:1788201258.538442:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442651520:192.168.201.130@tcp:2:chmod.0 00000100:00100000:3.0:1788201258.538459:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442651392:192.168.201.130@tcp:2:chmod.0 00000080:00200000:3.0:1788201258.538513:0:14316:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f6:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201258.538529:0:14316:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-5:[0x200000bd1:0x3f6:0x0](ffff9023449a42c8) current size 0, valid attrs 10000041, mode 8180, uid 1117589504, gid -28637, new size 0, atime 1788201242.000000000, mtime 1788201242.000000000, ctime 1788201242.000000000, rc 0 00000080:00200000:2.0:1788201258.566399:0:14315:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f5:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201258.566427:0:14315:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-4:[0x200000bd1:0x3f5:0x0](ffff90234497aa08) current size 0, valid attrs 10000041, mode 8180, uid 1436831744, gid -28637, new size 0, atime 1788201242.000000000, mtime 1788201242.000000000, ctime 1788201242.000000000, rc 0 00000080:00200000:3.0:1788201258.566430:0:14317:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f7:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201258.566445:0:14317:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-6:[0x200000bd1:0x3f7:0x0](ffff9023449a3a88) current size 0, valid attrs 10000041, mode 8180, uid 1204969472, gid -28637, new size 0, atime 1788201242.000000000, mtime 1788201242.000000000, ctime 1788201242.000000000, rc 0 00000080:00200000:2.0:1788201258.587374:0:14312:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f2:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201258.587405:0:14312:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-1:[0x200000bd1:0x3f2:0x0](ffff90234497d348) current size 0, valid attrs 10000041, mode 8180, uid 1117585408, gid -28637, new size 0, atime 1788201242.000000000, mtime 1788201242.000000000, ctime 1788201242.000000000, rc 0 00000080:00200000:2.0:1788201258.631839:0:14342:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201258.631866:0:14342:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201258.631917:0:14342:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:2.0:1788201258.650320:0:14345:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201258.650351:0:14345:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201258.650371:0:14345:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:2.0:1788201258.650387:0:14345:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:2.0:1788201258.650397:0:14345:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201258.650405:0:14345:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201258.650408:0:14345:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bc4 00000080:00010000:2.0:1788201258.650417:0:14345:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bc4 from it ffffa0b181be3b58 00010000:00010000:2.0:1788201258.650421:0:14345:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:2.0:1788201258.650431:0:14345:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:2.0:1788201258.650448:0:14345:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201258.650457:0:14345:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102b.replay-single, flags=24 00000080:00200000:2.0:1788201258.650463:0:14345:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448), inode mode 41ed mask 1 00000080:00200000:2.0:1788201258.650474:0:14345:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-2, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), flags=8 00000080:00200000:2.0:1788201258.650479:0:14345:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-2, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), intent=getattr 00000002:00010000:2.0:1788201258.650505:0:14345:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-2,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3f1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201258.650585:0:14345:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4db5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 11 type: IBT flags: 0x10000000000000 pid: 14345 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.650593:0:14345:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4db5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 11 type: IBT flags: 0x0 pid: 14345 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.650605:0:14345:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4db5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 11 type: IBT flags: 0x0 pid: 14345 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201258.650633:0:14345:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14345:1875064442652800:192.168.201.130@tcp:101:checkstat.0 00000100:00100000:2.0:1788201258.650730:0:14345:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5be40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201258.657363:0:14345:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14345:1875064442652800:192.168.201.130@tcp:101:checkstat.0 00010000:00010000:2.0:1788201258.657391:0:14345:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4db5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3a21 expref: -99 pid: 14345 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.657407:0:14345:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4db5 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3a21 expref: -99 pid: 14345 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.657415:0:14345:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4db5 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3a21 expref: -99 pid: 14345 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.657426:0:14345:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4db5 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3a21 expref: -99 pid: 14345 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.657437:0:14345:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4db5 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3a21 expref: -99 pid: 14345 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.657454:0:14345:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4db5 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3a21 expref: -99 pid: 14345 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.657462:0:14345:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4db5 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3a21 expref: -99 pid: 14345 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.657470:0:14345:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4db5 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3a21 expref: -99 pid: 14345 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201258.657481:0:14345:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c7800 x1875064442652800/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201274 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201258.657492:0:14345:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4db5 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3a21 expref: -99 pid: 14345 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201258.657506:0:14345:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4db5 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3a21 expref: -99 pid: 14345 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.657517:0:14345:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c4d7d lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d394f expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201258.657528:0:14345:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c4d7d lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d394f expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201258.657536:0:14345:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4db5 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3a21 expref: -99 pid: 14345 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.657563:0:14345:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4db5 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d3a21 expref: -99 pid: 14345 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.657570:0:14345:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c4db5 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d3a21 expref: -99 pid: 14345 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.657596:0:14345:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1788201258.657605:0:14345:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201258.657709:0:14345:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3f3:0x0](ffff9023449a3248): rc = 0 00000080:00200000:2.0:1788201258.657732:0:14345:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3f3:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201258.657743:0:14345:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f3:0x0](ffff9023449a3248) for lock 0x4605c7c97e1c4d7d 00000080:00010000:2.0:1788201258.657756:0:14345:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f3:0x0](ffff9023449a3248) 00000080:00010000:2.0:1788201258.657759:0:14345:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4d7d from it ffffa0b181be3b20 00010000:00010000:2.0:1788201258.657764:0:14345:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c4d7d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d394f expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201258.657772:0:14345:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c4d7d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d394f expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1788201258.657809:0:14345:0:(file.c:6283:ll_getattr_dentry()) START file file-2:[0x200000bd1:0x3f3:0x0](ffff9023449a3248), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201258.657827:0:14345:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3f3:0x0](ffff9023449a3248),name=file-2 00000002:00010000:2.0:1788201258.657846:0:14345:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3f3:0x0]) in obj [0x200000bd1:0x3f1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201258.657854:0:14345:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c4d7d lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d394f expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201258.657862:0:14345:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c4d7d lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d394f expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201258.657914:0:14345:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201258.657920:0:14345:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f3:0x0](ffff9023449a3248) 00000080:00010000:2.0:1788201258.657924:0:14345:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f3:0x0](ffff9023449a3248) for lock 0x4605c7c97e1c4d7d 00000080:00010000:2.0:1788201258.657929:0:14345:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4d7d from it ffffa0b181be3cb8 00010000:00010000:2.0:1788201258.657932:0:14345:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c4d7d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d394f expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201258.657941:0:14345:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c4d7d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d394f expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201258.657972:0:14345:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3f3:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201258.657980:0:14345:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3f3:0x0] 00000008:00010000:2.0:1788201258.658007:0:14345:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201258.658015:0:14345:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023533984b0, osc lock ffff9023500dcd10, flags 0x400001002 00010000:00010000:2.0:1788201258.658036:0:14345:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201258.658079:0:14345:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788201258.658120:0:14345:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788201258.658129:0:14345:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201258.658144:0:14345:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1788201258.658201:0:14345:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788201258.658215:0:14345:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x3f3:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201258.658222:0:14345:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_release():245 00000020:00010000:2.0:1788201258.658236:0:14345:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201258.658240:0:14345:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023533984b0 (ffff902350b9d758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201258.658245:0:14345:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201258.658248:0:14345:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023533984b0 (ffff902350b9d638/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201258.658278:0:14345:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f3:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201258.664529:0:14346:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201258.667481:0:14346:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201258.667521:0:14346:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:2.0:1788201258.667541:0:14346:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:2.0:1788201258.667560:0:14346:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201258.667569:0:14346:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201258.667573:0:14346:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bc4 00000080:00010000:2.0:1788201258.667583:0:14346:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bc4 from it ffffa0b181ca3b58 00010000:00010000:2.0:1788201258.667588:0:14346:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:2.0:1788201258.667598:0:14346:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:2.0:1788201258.667620:0:14346:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201258.667632:0:14346:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102b.replay-single, flags=24 00000080:00200000:2.0:1788201258.667641:0:14346:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448), inode mode 41ed mask 1 00000080:00200000:2.0:1788201258.667768:0:14346:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-3, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), flags=8 00000080:00200000:2.0:1788201258.667774:0:14346:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-3, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), intent=getattr 00000002:00010000:2.0:1788201258.667797:0:14346:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-3,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3f1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201258.667857:0:14346:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 14346 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.667865:0:14346:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14346 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.667912:0:14346:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14346 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201258.667947:0:14346:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14346:1875064442653184:192.168.201.130@tcp:101:checkstat.0 00000100:00100000:2.0:1788201258.668046:0:14346:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bf00 going to sleep for 11 seconds 00000100:00100000:2.0:1788201258.674152:0:14346:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14346:1875064442653184:192.168.201.130@tcp:101:checkstat.0 00010000:00010000:2.0:1788201258.674192:0:14346:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.674214:0:14346:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.674223:0:14346:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.674243:0:14346:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.674261:0:14346:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.674276:0:14346:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.674284:0:14346:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.674291:0:14346:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201258.674304:0:14346:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c7800 x1875064442653184/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201274 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201258.674316:0:14346:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201258.674330:0:14346:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.674345:0:14346:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4d84 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d396b expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201258.674359:0:14346:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4d84 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d396b expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201258.674373:0:14346:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.674381:0:14346:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.674388:0:14346:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.674417:0:14346:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1788201258.674485:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201258.674494:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.674503:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351685a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201258.674519:0:13086:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.674528:0:13086:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.674538:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 3/0,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.674566:0:13086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4d84 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d396b expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201258.674613:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 2/0,0 mode: --/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.674621:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201258.674662:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.674669:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4dc3 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3a3d expref: -99 pid: 14346 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.674685:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201258.674687:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201258.674689:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201258.674708:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442653312:192.168.201.130@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1788201258.674820:0:14346:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201258.675020:0:14346:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3f4:0x0](ffff9023449a1988): rc = 0 00000080:00200000:2.0:1788201258.675045:0:14346:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3f4:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201258.675057:0:14346:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f4:0x0](ffff9023449a1988) for lock 0x4605c7c97e1c4d84 00000080:00010000:2.0:1788201258.675074:0:14346:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f4:0x0](ffff9023449a1988) 00000080:00010000:2.0:1788201258.675078:0:14346:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4d84 from it ffffa0b181ca3b20 00010000:00010000:2.0:1788201258.675081:0:14346:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4d84 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d396b expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201258.675092:0:14346:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4d84 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d396b expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1788201258.675131:0:14346:0:(file.c:6283:ll_getattr_dentry()) START file file-3:[0x200000bd1:0x3f4:0x0](ffff9023449a1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201258.675140:0:14346:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3f4:0x0](ffff9023449a1988),name=file-3 00000002:00010000:2.0:1788201258.675163:0:14346:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3f4:0x0]) in obj [0x200000bd1:0x3f1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201258.675172:0:14346:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4d84 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d396b expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201258.675182:0:14346:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4d84 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d396b expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201258.675189:0:14346:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201258.675194:0:14346:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f4:0x0](ffff9023449a1988) 00000080:00010000:2.0:1788201258.675197:0:14346:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f4:0x0](ffff9023449a1988) for lock 0x4605c7c97e1c4d84 00000080:00010000:2.0:1788201258.675201:0:14346:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4d84 from it ffffa0b181ca3cb8 00010000:00010000:2.0:1788201258.675205:0:14346:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4d84 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d396b expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201258.675212:0:14346:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4d84 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d396b expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201258.675241:0:14346:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3f4:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201258.675253:0:14346:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3f4:0x0] 00000008:00010000:2.0:1788201258.675285:0:14346:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201258.675290:0:14346:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023533984b0, osc lock ffff9023500dcd10, flags 0x400001002 00010000:00010000:2.0:1788201258.675316:0:14346:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201258.675330:0:14346:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788201258.675343:0:14346:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788201258.675347:0:14346:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201258.675361:0:14346:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1788201258.675372:0:14346:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788201258.675386:0:14346:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x3f4:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201258.675391:0:14346:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_release():245 00000020:00010000:2.0:1788201258.675394:0:14346:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201258.675397:0:14346:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023533984b0 (ffff902350b9d758/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201258.675401:0:14346:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc4000 (ffff902350b9d638/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201258.675403:0:14346:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023533984b0 (ffff902350b9d638/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201258.675426:0:14346:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3f4:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1788201258.677776:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442653312:192.168.201.130@tcp:103:ldlm_bl.0 00000080:00200000:3.0:1788201258.697032:0:14347:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201258.697082:0:14347:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201258.697108:0:14347:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:3.0:1788201258.697128:0:14347:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:3.0:1788201258.697138:0:14347:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201258.697148:0:14347:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201258.697152:0:14347:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bc4 00000080:00010000:3.0:1788201258.697161:0:14347:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bc4 from it ffffa0b181cb3b58 00010000:00010000:3.0:1788201258.697165:0:14347:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:3.0:1788201258.697176:0:14347:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:3.0:1788201258.697196:0:14347:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201258.697207:0:14347:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102b.replay-single, flags=24 00000080:00200000:3.0:1788201258.697214:0:14347:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448), inode mode 41ed mask 1 00000080:00200000:3.0:1788201258.697229:0:14347:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-4, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), flags=8 00000080:00200000:3.0:1788201258.697234:0:14347:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-4, dir=[0x200000bd1:0x3f1:0x0](ffff90234497f448), intent=getattr 00000002:00010000:3.0:1788201258.697258:0:14347:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-4,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3f1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201258.697326:0:14347:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f000/0x4605c7c97e1c4dca lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 14347 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.697335:0:14347:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f000/0x4605c7c97e1c4dca lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14347 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.697348:0:14347:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f000/0x4605c7c97e1c4dca lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14347 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201258.697383:0:14347:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14347:1875064442653440:192.168.201.130@tcp:101:checkstat.0 00000100:00100000:3.0:1788201258.702210:0:14347:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436a6540 going to sleep for 11 seconds 00010000:00010000:2.0:1788201258.710709:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201258.710719:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f000/0x4605c7c97e1c4dca lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3a59 expref: -99 pid: 14347 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.710728:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351f9f000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201258.710733:0:5599:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f000/0x4605c7c97e1c4dca lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3a59 expref: -99 pid: 14347 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.710742:0:5599:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f000/0x4605c7c97e1c4dca lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3a59 expref: -99 pid: 14347 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.710751:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f000/0x4605c7c97e1c4dca lrc: 3/0,0 mode: PR/PR res: [0x200000bd1:0x3f5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d3a59 expref: -99 pid: 14347 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.710768:0:5599:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c4d8b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f5:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3987 expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201258.710825:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f000/0x4605c7c97e1c4dca lrc: 2/0,0 mode: --/PR res: [0x200000bd1:0x3f5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3a59 expref: -99 pid: 14347 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.710832:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201258.710865:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f000/0x4605c7c97e1c4dca lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3a59 expref: -99 pid: 14347 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.710872:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f000/0x4605c7c97e1c4dca lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f5:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3a59 expref: -99 pid: 14347 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.710925:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201258.710927:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201258.710929:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201258.711224:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442653568:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201258.740602:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442653568:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201258.753618:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201258.753626:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c4dd1 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3a75 expref: -99 pid: 14348 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.753646:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345803000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201258.753651:0:13084:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c4dd1 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3a75 expref: -99 pid: 14348 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.753659:0:13084:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c4dd1 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3a75 expref: -99 pid: 14348 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.753673:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c4dd1 lrc: 3/0,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d3a75 expref: -99 pid: 14348 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.753689:0:13084:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4d92 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d39a3 expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201258.753723:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c4dd1 lrc: 2/0,0 mode: --/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3a75 expref: -99 pid: 14348 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.753731:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201258.753776:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c4dd1 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3a75 expref: -99 pid: 14348 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.753784:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c4dd1 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f6:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3a75 expref: -99 pid: 14348 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.753799:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201258.753801:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201258.753803:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201258.753824:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201258.753840:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442653824:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201258.756871:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442653824:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201258.792811:0:14324:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c4de6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.794475:0:14324:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c4de6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201258.794596:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442654336:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:3.0:1788201258.821167:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201258.821180:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c4ddf lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3ac2 expref: -99 pid: 14350 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.821196:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839c800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201258.821202:0:13086:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c4ddf lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3ac2 expref: -99 pid: 14350 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.821212:0:13086:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c4ddf lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3ac2 expref: -99 pid: 14350 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.821221:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c4ddf lrc: 3/0,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d3ac2 expref: -99 pid: 14350 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.821242:0:13086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c4da0 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d39db expref: -99 pid: 14343 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201258.821285:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c4ddf lrc: 2/0,0 mode: --/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3ac2 expref: -99 pid: 14350 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.821292:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201258.821325:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c4ddf lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3ac2 expref: -99 pid: 14350 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.821333:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c4ddf lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f8:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3ac2 expref: -99 pid: 14350 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.821349:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201258.821351:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201258.821353:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201258.821497:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442654464:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201258.828579:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c4de6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.828605:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c4de6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.828615:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c4de6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.828624:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c4de6 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.828635:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c4de6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.828643:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351684600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201258.828655:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c4de6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.828664:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c4de6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.828673:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c4de6 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x20e400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.828682:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201258.828691:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c4de6 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201258.828698:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c4de6 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 14324 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201258.828807:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442654336:192.168.201.130@tcp:63:ll_sa.0 00000100:00100000:3.0:1788201258.836786:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442654464:192.168.201.130@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1788201258.839221:0:14351:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201258.839254:0:14351:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201258.839275:0:14351:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:2.0:1788201258.839294:0:14351:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:2.0:1788201258.839304:0:14351:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201258.839312:0:14351:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201258.839316:0:14351:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bc4 00000080:00010000:2.0:1788201258.839324:0:14351:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bc4 from it ffffa0b181cb3b58 00010000:00010000:2.0:1788201258.839328:0:14351:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:2.0:1788201258.839338:0:14351:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:2.0:1788201258.839354:0:14351:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201258.839364:0:14351:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102b.replay-single, flags=8 00000080:00200008:2.0:1788201258.839383:0:14351:0:(file.c:6283:ll_getattr_dentry()) START file d102b.replay-single:[0x200000bd1:0x3f1:0x0](ffff90234497f448), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201258.839388:0:14351:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448),name=d102b.replay-single 00000002:00010000:2.0:1788201258.839400:0:14351:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3f1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201258.839406:0:14351:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 30 res 8589937617/1009 (0 0) 00010000:00010000:2.0:1788201258.839451:0:14351:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 14351 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.839459:0:14351:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14351 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.839470:0:14351:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14351 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201258.839498:0:14351:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14351:1875064442654592:192.168.201.130@tcp:101:rm.0 00000100:00100000:2.0:1788201258.839600:0:14351:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bc00 going to sleep for 11 seconds 00000100:00100000:2.0:1788201258.843125:0:14351:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14351:1875064442654592:192.168.201.130@tcp:101:rm.0 00010000:00010000:2.0:1788201258.843232:0:14351:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ad0 expref: -99 pid: 14351 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.843250:0:14351:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ad0 expref: -99 pid: 14351 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.843257:0:14351:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ad0 expref: -99 pid: 14351 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.843268:0:14351:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3ad0 expref: -99 pid: 14351 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.843278:0:14351:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ad0 expref: -99 pid: 14351 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.843292:0:14351:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3ad0 expref: -99 pid: 14351 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.843301:0:14351:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3ad0 expref: -99 pid: 14351 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.843308:0:14351:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3ad0 expref: -99 pid: 14351 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201258.843320:0:14351:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c5880 x1875064442654592/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201274 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201258.843332:0:14351:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ad0 expref: -99 pid: 14351 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.843341:0:14351:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1009 (0 0) 00000080:00200000:2.0:1788201258.843349:0:14351:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201258.843442:0:14351:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x3f1:0x0], updating i_size 7680 00000080:00010000:2.0:1788201258.843450:0:14351:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f1:0x0](ffff90234497f448) for lock 0x4605c7c97e1c4ded 00000080:00010000:2.0:1788201258.843459:0:14351:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f1:0x0](ffff90234497f448) 00000080:00010000:2.0:1788201258.843463:0:14351:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4ded from it ffffa0b181cb3cb8 00010000:00010000:2.0:1788201258.843467:0:14351:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3ad0 expref: -99 pid: 14351 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.843475:0:14351:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ad0 expref: -99 pid: 14351 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201258.843584:0:14351:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201258.843601:0:14351:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201258.843612:0:14351:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:2.0:1788201258.843623:0:14351:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:2.0:1788201258.843630:0:14351:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201258.843635:0:14351:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201258.843638:0:14351:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4bc4 00000080:00010000:2.0:1788201258.843642:0:14351:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4bc4 from it ffffa0b181cb3bf8 00010000:00010000:2.0:1788201258.843645:0:14351:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:2.0:1788201258.843658:0:14351:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c4bc4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3566 expref: -99 pid: 14290 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:2.0:1788201258.843667:0:14351:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201258.843674:0:14351:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102b.replay-single, flags=266 00000080:00200000:2.0:1788201258.843679:0:14351:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448), inode mode 41ed mask 44 00000080:00200008:2.0:1788201258.843687:0:14351:0:(file.c:1090:ll_file_open()) START file d102b.replay-single:[0x200000bd1:0x3f1:0x0](ffff90234497f448), flags 704400 00000002:00010000:2.0:1788201258.843704:0:14351:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3f1:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:2.0:1788201258.844017:0:14351:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4df4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 14351 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.844026:0:14351:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4df4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14351 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201258.844039:0:14351:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201258.844047:0:14351:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c5880 x1875064442654848/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201258.844056:0:14351:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4df4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14351 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201258.844078:0:14351:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14351:1875064442654848:192.168.201.130@tcp:101:rm.0 00000100:00100000:2.0:1788201258.844339:0:14351:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bc00 going to sleep for 11 seconds 00000100:00100000:2.0:1788201258.851987:0:14351:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14351:1875064442654848:192.168.201.130@tcp:101:rm.0 00010000:00010000:2.0:1788201258.852013:0:14351:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4df4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14351 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.852025:0:14351:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4df4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 14351 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.852032:0:14351:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4df4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 14351 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.852038:0:14351:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4df4 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 14351 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.852046:0:14351:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4df4 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 14351 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.852053:0:14351:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351684c00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201258.852057:0:14351:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4df4 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 14351 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.852063:0:14351:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4df4 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 14351 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.852070:0:14351:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4df4 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 14351 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.852076:0:14351:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201258.852082:0:14351:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4df4 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 14351 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201258.852089:0:14351:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4df4 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 14351 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201258.852107:0:14351:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9023458c5880 x1875064442654848/t8589938685(8589938685) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201274 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201258.852120:0:14351:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458c5880 x1875064442654848/t8589938685(8589938685) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201274 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201258.852144:0:14351:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201258.852166:0:14351:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x3f1:0x0], updating i_size 7680 00000002:00100000:2.0:1788201258.852188:0:14351:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458c5880 x1875064442654848/t8589938685(8589938685) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201274 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788201258.852280:0:14351:0:(file.c:6283:ll_getattr_dentry()) START file d102b.replay-single:[0x200000bd1:0x3f1:0x0](ffff90234497f448), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201258.852288:0:14351:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448),name=d102b.replay-single 00000002:00010000:2.0:1788201258.852309:0:14351:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3f1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201258.852326:0:14351:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ad0 expref: -99 pid: 14351 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201258.852340:0:14351:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ad0 expref: -99 pid: 14351 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201258.852347:0:14351:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201258.852353:0:14351:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f1:0x0](ffff90234497f448) 00000080:00010000:2.0:1788201258.852357:0:14351:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f1:0x0](ffff90234497f448) for lock 0x4605c7c97e1c4ded 00000080:00010000:2.0:1788201258.852362:0:14351:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4ded from it ffffa0b181cb3cd8 00010000:00010000:2.0:1788201258.852366:0:14351:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3ad0 expref: -99 pid: 14351 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201258.852386:0:14351:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ad0 expref: -99 pid: 14351 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201258.852473:0:14351:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000bd1:0x3f1:0x0](ffff90234497f448) pos/size0/7680 32bit_api 0 00000002:00010000:2.0:1788201258.852488:0:14351:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3f1:0x0]) in obj [0x200000bd1:0x3f1:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788201258.852495:0:14351:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ad0 expref: -99 pid: 14351 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201258.852503:0:14351:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c4ded lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ad0 expref: -99 pid: 14351 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1788201258.853662:0:14351:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7480 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14351:1875064442654976:192.168.201.130@tcp:37:rm.0 00000100:00100000:2.0:1788201258.853762:0:14351:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bc00 going to sleep for 11 seconds 00010000:00010000:2.0:1788201259.468966:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201259.840970:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201259.841015:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201259.841052:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458224512 00000100:00100000:2.0:1788201259.841065:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458224512 00000100:00100000:2.0:1788201259.841123:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458224768 00000100:00100000:2.0:1788201259.841127:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458224768 00000100:00100000:2.0:1788201259.841146:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1319 00000100:00100000:2.0:1788201259.841157:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c4000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13438:x1875064458224512:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201259.841180:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000000bc9abec/0x4605c7c97e1c4c50 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x419:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d3757 expref: -99 pid: 14312 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201259.841283:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201259.841291:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c4000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13438:x1875064458224512:12345-192.168.201.130@tcp:104: Request processed in 136us (339us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201259.841298:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1319 00000100:00100000:2.0:1788201259.841317:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1320 00000100:00100000:2.0:1788201259.841323:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c5180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13447:x1875064458224768:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201259.841332:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c4d6f lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d3902 expref: -99 pid: 14317 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201259.841369:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:2.0:1788201259.841375:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c5180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13447:x1875064458224768:12345-192.168.201.130@tcp:104: Request processed in 52us (360us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201259.841381:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1320 00010000:00010000:2.0:1788201259.841743:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201259.841749:0:5598:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c4d6f lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d3902 expref: -99 pid: 14317 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201259.841758:0:5598:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351684000 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201259.841762:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c4d6f lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d3902 expref: -99 pid: 14317 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201259.841819:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c4d6f lrc: 2/0,0 mode: --/PR res: [0x240000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3902 expref: -99 pid: 14317 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201259.841825:0:5598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201259.841844:0:5598:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c4d6f lrc: 1/0,0 mode: --/PR res: [0x240000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3902 expref: -99 pid: 14317 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201259.841852:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c4d6f lrc: 0/0,0 mode: --/PR res: [0x240000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3902 expref: -99 pid: 14317 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201259.841868:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201259.841870:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201259.841871:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201259.841901:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442658176:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201259.860540:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442658176:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201260.483683:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201261.505119:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201262.533555:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201263.494976:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442658432:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201263.495096:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442658688:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201263.495131:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788201263.495137:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442658304:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201263.495164:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442658560:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201263.502732:0:2361:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0000-osc-ffff902350a11000: committing for last_committed 4294968849 gen 2 00000100:00100000:3.0:1788201263.502771:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442658560:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201263.502784:0:2361:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff902350a11000: committing for last_committed 4294968850 gen 2 00000100:00100000:3.0:1788201263.502800:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442658688:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201263.506726:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442658304:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201263.506747:0:2361:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 8589938696 gen 2 00000100:00100000:2.0:1788201263.506786:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442658432:192.168.201.130@tcp:400:kworker.0 00010000:00010000:3.0:1788201263.554210:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201264.578100:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201264.906489:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201264.906548:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201264.906585:0:5590:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1788201264.906605:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458226304 00000100:00100000:2.0:1788201264.906615:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458226304 00000100:00100000:2.0:1788201264.906663:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458226432 00000100:00100000:2.0:1788201264.906666:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458226432 00000100:00100000:2.0:1788201264.906681:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1321 00000100:00100000:2.0:1788201264.906691:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c5500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+5:13441:x1875064458226304:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201264.906709:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c128faeb/0x4605c7c97e1c4d22 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d3899 expref: -99 pid: 14316 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201264.906790:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201264.906798:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c5500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+5:13441:x1875064458226304:12345-192.168.201.130@tcp:104: Request processed in 108us (323us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201264.906798:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1322 00000100:00100000:2.0:1788201264.906805:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1321 00000100:00100000:3.0:1788201264.906807:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902342dfa450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13441:x1875064458226432:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201264.906815:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000750e72d7/0x4605c7c97e1c4d1b lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d37ff expref: -99 pid: 14320 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201264.906846:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1788201264.906852:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902342dfa450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13441:x1875064458226432:12345-192.168.201.130@tcp:104: Request processed in 48us (314us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201264.906858:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1322 00010000:00010000:2.0:1788201264.906952:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1788201264.906959:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c128faeb/0x4605c7c97e1c4d22 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d3899 expref: -99 pid: 14316 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201264.906969:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db0200 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201264.906976:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c128faeb/0x4605c7c97e1c4d22 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d3899 expref: -99 pid: 14316 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201264.914914:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c128faeb/0x4605c7c97e1c4d22 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3899 expref: -99 pid: 14316 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201264.914934:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201264.914970:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c128faeb/0x4605c7c97e1c4d22 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3899 expref: -99 pid: 14316 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201264.914979:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c128faeb/0x4605c7c97e1c4d22 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x41b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3899 expref: -99 pid: 14316 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201264.915036:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442658816:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201264.915605:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201264.915609:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201264.915611:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201264.918834:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201264.918865:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201264.918907:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458226816 00000100:00100000:2.0:1788201264.918916:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458226816 00000100:00100000:2.0:1788201264.918959:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458226944 00000100:00100000:2.0:1788201264.918963:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458226944 00000100:00100000:2.0:1788201264.918979:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1323 00000100:00100000:2.0:1788201264.918988:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c7480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:13442:x1875064458226816:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201264.919006:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201264.919080:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201264.919087:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c7480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:13442:x1875064458226816:12345-192.168.201.130@tcp:104: Request processed in 100us (261us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201264.919095:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1323 00000100:00100000:2.0:1788201264.919112:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1324 00000100:00100000:2.0:1788201264.919118:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c4a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6706:x1875064458226944:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201264.919126:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c4c81 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x41b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d37ea expref: -99 pid: 14315 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201264.919151:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:2.0:1788201264.919156:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c4a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:6706:x1875064458226944:12345-192.168.201.130@tcp:104: Request processed in 38us (291us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201264.919161:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1324 00010000:00010000:2.0:1788201264.919176:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1788201264.919182:0:5598:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201264.919191:0:5598:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351685200 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201264.919196:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201264.919272:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 2/0,0 mode: --/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201264.919280:0:5598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201264.919301:0:5598:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 1/0,0 mode: --/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201264.919309:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081827299/0x4605c7c97e1c4c5e lrc: 0/0,0 mode: --/PR res: [0x240000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d377a expref: -99 pid: 14313 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201264.919468:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201264.919475:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c4c81 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x41b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d37ea expref: -99 pid: 14315 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201264.919483:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839d400 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201264.919487:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c4c81 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x41b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d37ea expref: -99 pid: 14315 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201264.919538:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c4c81 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x41b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d37ea expref: -99 pid: 14315 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201264.919545:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201264.919552:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c4c81 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x41b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d37ea expref: -99 pid: 14315 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201264.919560:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c4c81 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x41b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d37ea expref: -99 pid: 14315 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201264.919573:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201264.919575:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201264.919577:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201264.919774:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201264.919777:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201264.919779:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201264.919795:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442659072:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201264.919859:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201264.919865:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442659200:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201264.920298:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201264.920365:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201264.920843:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442658816:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201264.920871:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458227072 00000100:00100000:2.0:1788201264.920893:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458227072 00000100:00100000:2.0:1788201264.920921:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458227200 00000100:00100000:2.0:1788201264.920924:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458227200 00000100:00100000:2.0:1788201264.920937:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1325 00000100:00100000:2.0:1788201264.920945:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c6a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:10484:x1875064458227072:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201264.920959:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201264.921013:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201264.921020:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c6a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:10484:x1875064458227072:12345-192.168.201.130@tcp:104: Request processed in 75us (723us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201264.921027:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1325 00000100:00100000:2.0:1788201264.921042:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1326 00000100:00100000:2.0:1788201264.921049:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c6d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13438:x1875064458227200:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201264.921057:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c4ce3 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d37f8 expref: -99 pid: 14318 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201264.921084:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:2.0:1788201264.921090:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c6d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13438:x1875064458227200:12345-192.168.201.130@tcp:104: Request processed in 42us (724us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201264.921096:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1326 00010000:00010000:2.0:1788201264.921139:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1788201264.921144:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201264.921152:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db1600 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201264.921158:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201264.921219:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 2/0,0 mode: --/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201264.921226:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201264.921241:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 1/0,0 mode: --/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201264.921248:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000038133334/0x4605c7c97e1c4c6c lrc: 0/0,0 mode: --/PR res: [0x280000400:0x41a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d37a4 expref: -99 pid: 14314 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1788201264.921276:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442659328:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201264.921334:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201264.921338:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c4ce3 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d37f8 expref: -99 pid: 14318 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201264.921345:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345803a00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201264.921348:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c4ce3 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d37f8 expref: -99 pid: 14318 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201264.921381:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c4ce3 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d37f8 expref: -99 pid: 14318 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201264.921387:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201264.921396:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c4ce3 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d37f8 expref: -99 pid: 14318 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201264.921404:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c4ce3 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x41c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d37f8 expref: -99 pid: 14318 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1788201264.921445:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442659456:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201264.922862:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201264.922867:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201264.922869:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201264.925506:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442659200:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201264.925521:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442659072:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201264.934995:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442659456:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201264.935036:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442659328:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201265.601158:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201267.179051:0:14915:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235312ad80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201267.189944:0:14915:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1500 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14915:1875064442659584:192.168.201.130@tcp:101:lfs.0 00010000:00010000:3.0:1788201267.189972:0:14915:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e02 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c0b expref: -99 pid: 14915 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.189985:0:14915:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e02 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c0b expref: -99 pid: 14915 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.189997:0:14915:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e02 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3c0b expref: -99 pid: 14915 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.190013:0:14915:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e02 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c0b expref: -99 pid: 14915 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.190026:0:14915:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e02 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3c0b expref: -99 pid: 14915 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.190035:0:14915:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e02 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3c0b expref: -99 pid: 14915 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.190042:0:14915:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e02 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3c0b expref: -99 pid: 14915 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201267.190053:0:14915:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023488b1500 x1875064442659584/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201283 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201267.190067:0:14915:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e02 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c0b expref: -99 pid: 14915 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.190079:0:14915:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201267.190086:0:14915:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201267.190113:0:14915:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:3.0:1788201267.190118:0:14915:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4e02 00000080:00010000:3.0:1788201267.190123:0:14915:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201267.190126:0:14915:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4e02 from it ffffa0b181d13b58 00010000:00010000:3.0:1788201267.190129:0:14915:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e02 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3c0b expref: -99 pid: 14915 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.190141:0:14915:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e02 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c0b expref: -99 pid: 14915 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201267.190165:0:14915:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201267.190233:0:14915:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d102c.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=13 00000080:00200000:3.0:1788201267.190238:0:14915:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d102c.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=getattr 00000002:00010000:3.0:1788201267.190257:0:14915:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d102c.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201267.190308:0:14915:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ea00/0x4605c7c97e1c4e09 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 14915 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.190315:0:14915:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ea00/0x4605c7c97e1c4e09 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14915 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.190324:0:14915:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ea00/0x4605c7c97e1c4e09 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14915 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201267.190344:0:14915:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14915:1875064442659712:192.168.201.130@tcp:101:lfs.0 00000100:00100000:3.0:1788201267.190519:0:14915:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358668d80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201267.229387:0:14915:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b2680 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14915:1875064442659968:192.168.201.130@tcp:36:lfs.0 00000080:00200000:3.0:1788201267.229454:0:14915:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148) 00000080:00200000:3.0:1788201267.229461:0:14915:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x3f9:0x0], updating i_size 512 00000080:00200008:3.0:1788201267.229494:0:14915:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 300000 00000100:00100000:2.0:1788201267.296519:0:14923:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b0380 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14923:1875064442660096:192.168.201.130@tcp:101:lfs.0 00010000:00010000:2.0:1788201267.296562:0:14923:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.296577:0:14923:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.296590:0:14923:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.296601:0:14923:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.296615:0:14923:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.296624:0:14923:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.296631:0:14923:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201267.296644:0:14923:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023488b0380 x1875064442660096/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201283 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201267.296659:0:14923:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.296671:0:14923:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201267.296678:0:14923:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201267.296708:0:14923:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:2.0:1788201267.296714:0:14923:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4e10 00000080:00010000:2.0:1788201267.296720:0:14923:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201267.296722:0:14923:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4e10 from it ffffa0b181bf3bf8 00010000:00010000:2.0:1788201267.296726:0:14923:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.296734:0:14923:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201267.296765:0:14923:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201267.296806:0:14923:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=d102c.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), file ffff9023584e3200, open_flags 18800, mode 0 opened 0 00000080:00200000:2.0:1788201267.296813:0:14923:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d102c.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=open 00000002:00010000:2.0:1788201267.296836:0:14923:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d102c.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0304001 00010000:00010000:2.0:1788201267.296913:0:14923:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4e17 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 14923 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.296920:0:14923:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4e17 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14923 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201267.296933:0:14923:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201267.296939:0:14923:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c5880 x1875064442660352/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201267.296949:0:14923:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4e17 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14923 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201267.296971:0:14923:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14923:1875064442660352:192.168.201.130@tcp:101:lfs.0 00000100:00100000:2.0:1788201267.297269:0:14923:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b540 going to sleep for 11 seconds 00000100:00100000:3.0:1788201267.299744:0:14923:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14923:1875064442660352:192.168.201.130@tcp:101:lfs.0 00010000:00010000:3.0:1788201267.299767:0:14923:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4e17 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14923 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.299792:0:14923:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4e17 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 14923 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.299801:0:14923:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4e17 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 14923 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.299808:0:14923:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4e17 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 14923 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.299816:0:14923:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4e17 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 14923 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.299822:0:14923:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351685a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201267.299827:0:14923:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4e17 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 14923 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.299843:0:14923:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4e17 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 14923 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.299851:0:14923:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4e17 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 14923 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.299859:0:14923:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201267.299865:0:14923:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4e17 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 14923 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.299871:0:14923:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c4e17 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 14923 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201267.300044:0:14923:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9023458c5880 x1875064442660352/t8589938698(8589938698) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/608 e 0 to 0 dl 1788201283 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201267.300057:0:14923:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458c5880 x1875064442660352/t8589938698(8589938698) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/608 e 0 to 0 dl 1788201283 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201267.300071:0:14923:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201267.300094:0:14923:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x3f9:0x0], updating i_size 512 00000080:00200000:3.0:1788201267.300097:0:14923:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3f9:0x0](ffff902344979148): rc = 0 00000080:00010000:3.0:1788201267.300101:0:14923:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f9:0x0](ffff902344979148) for lock 0x0 00000080:00200008:3.0:1788201267.300126:0:14923:0:(file.c:1090:ll_file_open()) START file d102c.replay-single:[0x200000bd1:0x3f9:0x0](ffff902344979148), flags 304000 00000002:00100000:3.0:1788201267.300145:0:14923:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458c5880 x1875064442660352/t8589938698(8589938698) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/608 e 0 to 0 dl 1788201283 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201267.300208:0:14923:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 44 00000080:00200000:3.0:1788201267.300483:0:14923:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201267.300500:0:14923:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201267.300515:0:14923:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201267.300527:0:14923:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201267.300547:0:14923:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201267.300552:0:14923:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201267.300556:0:14923:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4e10 00000080:00010000:3.0:1788201267.300560:0:14923:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4e10 from it ffffa0b181bf3bf8 00010000:00010000:3.0:1788201267.300564:0:14923:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201267.300799:0:14923:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201267.300820:0:14923:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201267.300843:0:14923:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=d102c.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), file ffff902346275200, open_flags 28800, mode 0 opened 0 00000080:00200000:3.0:1788201267.300850:0:14923:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d102c.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=open 00000002:00010000:3.0:1788201267.300864:0:14923:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d102c.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0504001 00010000:00010000:3.0:1788201267.300928:0:14923:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4e1e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 14923 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.300936:0:14923:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4e1e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14923 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201267.300948:0:14923:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201267.300955:0:14923:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90235268b100 x1875064442660608/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201267.300964:0:14923:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4e1e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14923 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201267.300985:0:14923:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14923:1875064442660608:192.168.201.130@tcp:101:lfs.0 00000100:00100000:3.0:1788201267.301157:0:14923:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358668d80 going to sleep for 11 seconds 00000080:00200000:2.0:1788201267.443839:0:14930:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201267.443870:0:14930:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201267.444046:0:14930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201267.444064:0:14930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201267.444072:0:14930:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201267.444082:0:14930:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201267.444086:0:14930:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4e10 00000080:00010000:2.0:1788201267.444094:0:14930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4e10 from it ffffa0b181c03bf8 00010000:00010000:2.0:1788201267.444098:0:14930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201267.444106:0:14930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201267.444123:0:14930:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201267.444131:0:14930:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102c.replay-single, flags=25 00000080:00200000:2.0:1788201267.444136:0:14930:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201267.444148:0:14930:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:2.0:1788201267.444226:0:14930:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file-2, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), file ffff90235877f680, open_flags 8941, mode 81b6 opened 0 00000080:00200000:2.0:1788201267.444237:0:14930:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-2, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), intent=open|creat 00000002:00010000:2.0:1788201267.444258:0:14930:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-2,[0x200000bd1:0x3fb:0x0]) in obj [0x200000bd1:0x3f9:0x0], intent: open|creat flags 0104502 00010000:00010000:2.0:1788201267.444312:0:14930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 14930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.444320:0:14930:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14930 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201267.444334:0:14930:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201267.444346:0:14930:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c7b80 x1875064442662656/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201267.444356:0:14930:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14930 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201267.444454:0:14930:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14930:1875064442662656:192.168.201.130@tcp:101:touch.0 00000100:00100000:2.0:1788201267.444652:0:14930:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5be40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201267.450199:0:14930:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14930:1875064442662656:192.168.201.130@tcp:101:touch.0 00010000:00010000:2.0:1788201267.450231:0:14930:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ce4 expref: -99 pid: 14930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.450253:0:14930:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ce4 expref: -99 pid: 14930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.450261:0:14930:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ce4 expref: -99 pid: 14930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.450271:0:14930:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3ce4 expref: -99 pid: 14930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.450282:0:14930:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ce4 expref: -99 pid: 14930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.450295:0:14930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3ce4 expref: -99 pid: 14930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.450303:0:14930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3ce4 expref: -99 pid: 14930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.450309:0:14930:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3ce4 expref: -99 pid: 14930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201267.450321:0:14930:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023458c7b80 x1875064442662656/t8589938705(8589938705) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201283 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201267.450334:0:14930:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458c7b80 x1875064442662656/t8589938705(8589938705) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201283 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201267.450346:0:14930:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ce4 expref: -99 pid: 14930 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201267.450359:0:14930:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ce4 expref: -99 pid: 14930 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.450380:0:14930:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1019 (0 0) 00000080:00200000:2.0:1788201267.450387:0:14930:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x3f9:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201267.450409:0:14930:0:(namei.c:1628:ll_create_it()) VFS Op:name=file-2, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), intent=open|creat 00000080:00200000:2.0:1788201267.450436:0:14930:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x3fb:0x0](ffff9023449aba88) 00000080:00200000:2.0:1788201267.450557:0:14930:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3fb:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788201267.450568:0:14930:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x3fb:0x0](ffff9023449aba88) 00000080:00010000:2.0:1788201267.450571:0:14930:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3fb:0x0](ffff9023449aba88) for lock 0x4605c7c97e1c4e3a 00000080:00010000:2.0:1788201267.450581:0:14930:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3fb:0x0](ffff9023449aba88) 00000080:00200008:2.0:1788201267.450593:0:14930:0:(file.c:1090:ll_file_open()) START file file-2:[0x200000bd1:0x3fb:0x0](ffff9023449aba88), flags 104501 00000002:00100000:2.0:1788201267.450613:0:14930:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458c7b80 x1875064442662656/t8589938705(8589938705) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201283 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788201267.450653:0:14930:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4e3a from it ffff902342d49060 00010000:00010000:2.0:1788201267.450658:0:14930:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3ce4 expref: -99 pid: 14930 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.450667:0:14930:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3ce4 expref: -99 pid: 14930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201267.450694:0:14930:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3fb:0x0](ffff9023449aba88), inode mode 81a4 mask 40 00000080:00200008:2.0:1788201267.450748:0:14930:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-2:[0x200000bd1:0x3fb:0x0](ffff9023449aba88) current size 0, valid attrs 20070, mode 9023, uid -2118107568, gid -24399, new size 18446744071973566999, atime 1788201267.000000000, mtime 1788201267.000000000, ctime 1788201267.000000000 00010000:00010000:2.0:1788201267.450768:0:14930:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d3ce4 expref: -99 pid: 14930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201267.450792:0:14930:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3fb:0x0]: losing layout lock 00010000:00010000:2.0:1788201267.450835:0:14930:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3ce4 expref: -99 pid: 14930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201267.450841:0:14930:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201267.450844:0:14930:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c4e3a lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3ce4 expref: -99 pid: 14930 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788201267.450866:0:14930:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201267.450872:0:14930:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c7800 x1875064442662912/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201267.451549:0:14930:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14930:1875064442662912:192.168.201.130@tcp:36:touch.0 00000100:00100000:2.0:1788201267.451639:0:14930:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b240 going to sleep for 11 seconds 00000100:00100000:2.0:1788201267.457187:0:14930:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14930:1875064442662912:192.168.201.130@tcp:36:touch.0 00000080:00200000:2.0:1788201267.457267:0:14930:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3fb:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:3.0:1788201267.464154:0:14930:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x3fb:0x0](ffff9023449aba88) 00000002:00010000:3.0:1788201267.464178:0:14930:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3fb:0x0]) in obj [0x200000bd1:0x3fb:0x0], intent: layout flags 0 00010000:00010000:3.0:1788201267.464244:0:14930:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e400/0x4605c7c97e1c4e41 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 14930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.464253:0:14930:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e400/0x4605c7c97e1c4e41 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 14930 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201267.464261:0:14930:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e400/0x4605c7c97e1c4e41 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 14930 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201267.464285:0:14930:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14930:1875064442663040:192.168.201.130@tcp:101:touch.0 00000100:00100000:3.0:1788201267.464469:0:14930:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358668d80 going to sleep for 11 seconds 00000080:00200000:3.0:1788201267.471147:0:14930:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3fb:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201267.471173:0:14930:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-2:[0x200000bd1:0x3fb:0x0](ffff9023449aba88) current size 0, valid attrs 100201f0, mode 9023, uid -2118107568, gid -24399, new size 18446744071973566999, atime 1788201267.000000000, mtime 1788201267.000000000, ctime 1788201267.000000000, rc 0 00000080:00200008:3.0:1788201267.471205:0:14930:0:(file.c:479:ll_file_release()) START file file-2:[0x200000bd1:0x3fb:0x0](ffff9023449aba88), flags 104001 00000002:00100000:3.0:1788201267.471255:0:14930:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023458c7b80 x1875064442662656/t8589938705(8589938705) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201283 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201267.471264:0:14930:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201267.471268:0:14930:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352688380 x1875064442663424/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201267.471284:0:14930:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14930:1875064442663424:192.168.201.130@tcp:35:touch.0 00000100:00100000:3.0:1788201267.471343:0:14930:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358668d80 going to sleep for 11 seconds 00000080:00200000:2.0:1788201267.568233:0:14932:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3fd:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201267.568259:0:14932:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-4:[0x200000bd1:0x3fd:0x0](ffff9023449b63c8) current size 0, valid attrs 100201f0, mode 9023, uid -2116731312, gid -24399, new size 18446744071973566999, atime 1788201267.000000000, mtime 1788201267.000000000, ctime 1788201267.000000000, rc 0 00000080:00200008:2.0:1788201267.568308:0:14932:0:(file.c:479:ll_file_release()) START file file-4:[0x200000bd1:0x3fd:0x0](ffff9023449b63c8), flags 104001 00000002:00100000:2.0:1788201267.568368:0:14932:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b1880 x1875064442664704/t8589938711(8589938711) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201283 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201267.568397:0:14932:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201267.568403:0:14932:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c5c00 x1875064442665472/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201267.568426:0:14932:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5c00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14932:1875064442665472:192.168.201.130@tcp:35:touch.0 00000100:00100000:2.0:1788201267.568717:0:14932:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b240 going to sleep for 11 seconds 00000080:00200000:2.0:1788201267.583622:0:14933:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201267.583654:0:14933:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201267.583674:0:14933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201267.583692:0:14933:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788201267.583701:0:14933:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201267.583707:0:14933:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201267.583711:0:14933:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4e10 00000080:00010000:2.0:1788201267.583718:0:14933:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4e10 from it ffffa0b180d4bbf8 00010000:00010000:2.0:1788201267.583722:0:14933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201267.583732:0:14933:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ce00/0x4605c7c97e1c4e10 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3c3c expref: -99 pid: 14923 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788201267.583754:0:14933:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201267.583761:0:14933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102c.replay-single, flags=25 00000080:00200000:2.0:1788201267.583767:0:14933:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201267.583780:0:14933:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:2.0:1788201267.583784:0:14933:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file-5, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), file ffff902350cc1200, open_flags 8941, mode 81b6 opened 0 00000080:00200000:2.0:1788201267.583791:0:14933:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-5, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), intent=open|creat 00000002:00010000:2.0:1788201267.583812:0:14933:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-5,[0x200000bd1:0x3fe:0x0]) in obj [0x200000bd1:0x3f9:0x0], intent: open|creat flags 0104502 00010000:00010000:2.0:1788201267.583866:0:14933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4e64 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 14933 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201267.584286:0:14933:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4e64 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14933 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201267.584307:0:14933:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201267.584323:0:14933:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c4380 x1875064442665728/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201267.584334:0:14933:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4e64 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14933 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201267.584358:0:14933:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4380 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14933:1875064442665728:192.168.201.130@tcp:101:touch.0 00000100:00100000:2.0:1788201267.584692:0:14933:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bc00 going to sleep for 11 seconds 00010000:00010000:3.0:1788201267.658123:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201269.701950:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201271.469874:0:14985:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00010000:00010000:3.0:1788201271.745619:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788201272.575685:0:15001:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201272.575714:0:15001:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201272.575728:0:15001:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1788201272.575782:0:15001:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 15001 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.575796:0:15001:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 15001 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.575806:0:15001:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 15001 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201272.575833:0:15001:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15001:1875064442670848:192.168.201.130@tcp:101:chmod.0 00000100:00100000:2.0:1788201272.575960:0:15001:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b900 going to sleep for 11 seconds 00000080:00200000:2.0:1788201272.576081:0:14999:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201272.576090:0:14999:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201272.576096:0:14999:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:2.0:1788201272.576113:0:14999:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4ebf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 6 type: IBT flags: 0x10000000000000 pid: 14999 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.576121:0:14999:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4ebf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 6 type: IBT flags: 0x0 pid: 14999 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.576128:0:14999:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4ebf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 6 type: IBT flags: 0x0 pid: 14999 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201272.576139:0:14999:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14999:1875064442670976:192.168.201.130@tcp:101:chmod.0 00000100:00100000:2.0:1788201272.576161:0:14999:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b780 going to sleep for 11 seconds 00000100:00100000:3.0:1788201272.587999:0:14997:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b0e00 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14997:1875064442670592:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201272.588028:0:14997:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 8 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.588041:0:14997:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 8 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.588059:0:14997:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 8 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.588071:0:14997:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 8 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.588085:0:14997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 8 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.588093:0:14997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 8 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.588100:0:14997:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 8 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201272.588112:0:14997:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023488b0e00 x1875064442670592/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201288 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201272.588127:0:14997:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 8 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.588140:0:14997:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201272.588149:0:14997:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201272.588180:0:14997:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:3.0:1788201272.588197:0:14997:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4eaa 00000080:00010000:3.0:1788201272.588201:0:14997:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201272.588205:0:14997:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4eaa from it ffffa0b181c8bb58 00010000:00010000:3.0:1788201272.588209:0:14997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 8 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.588218:0:14997:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 8 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201272.588370:0:14997:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201272.588383:0:14997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102c.replay-single, flags=25 00000080:00200000:3.0:1788201272.588392:0:14997:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:3.0:1788201272.588477:0:14997:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-2, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), flags=9 00000080:00200000:3.0:1788201272.588483:0:14997:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-2, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), intent=getattr 00000002:00010000:3.0:1788201272.588506:0:14997:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-2,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3f9:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201272.588729:0:14997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 14997 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.588738:0:14997:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14997 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.588749:0:14997:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 14997 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201272.588773:0:14997:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689f80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14997:1875064442671360:192.168.201.130@tcp:101:chmod.0 00000100:00100000:3.0:1788201272.588856:0:14997:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023587e5cc0 going to sleep for 11 seconds 00010000:00010000:3.0:1788201272.590117:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201272.590124:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4ea3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 7 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3e57 expref: -99 pid: 14996 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.590138:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345802a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201272.590145:0:13086:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4ea3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 7 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3e57 expref: -99 pid: 14996 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.590171:0:13086:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4ea3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 8 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3e57 expref: -99 pid: 14996 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.590183:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4ea3 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 7 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d3e57 expref: -99 pid: 14996 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.590215:0:13086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 7 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201272.590761:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4ea3 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 7 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3e57 expref: -99 pid: 14996 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.590770:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201272.590789:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4ea3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 7 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3e57 expref: -99 pid: 14996 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.590797:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4ea3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 7 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3e57 expref: -99 pid: 14996 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.590809:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201272.590810:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201272.590812:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201272.590836:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442671872:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201272.598271:0:14999:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14999:1875064442670976:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201272.598300:0:14999:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4ebf lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e8f expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.598322:0:14999:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4ebf lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e8f expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.598333:0:14999:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4ebf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e8f expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.598344:0:14999:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4ebf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e8f expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.598356:0:14999:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4ebf lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e8f expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.598363:0:14999:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4ebf lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e8f expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.598370:0:14999:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4ebf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e8f expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201272.598383:0:14999:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c6680 x1875064442670976/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201288 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201272.598394:0:14999:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4ebf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e8f expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.598410:0:14999:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 7 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201272.598422:0:14999:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201272.598430:0:14999:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4ebf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e8f expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.598436:0:14999:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4ebf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d3e8f expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.598444:0:14999:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c4ebf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d3e8f expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.598467:0:14999:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1788201272.598474:0:14999:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201272.598502:0:14999:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:2.0:1788201272.598511:0:14999:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4eaa 00000080:00010000:2.0:1788201272.598522:0:14999:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201272.598527:0:14999:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4eaa from it ffffa0b181d33b58 00010000:00010000:2.0:1788201272.598533:0:14999:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 7 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201272.598543:0:14999:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 7 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201272.598564:0:14999:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201272.598574:0:14999:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102c.replay-single, flags=25 00000080:00200000:2.0:1788201272.598581:0:14999:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201272.598593:0:14999:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-4, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), flags=9 00000080:00200000:2.0:1788201272.598600:0:14999:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-4, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), intent=getattr 00000002:00010000:2.0:1788201272.598621:0:14999:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-4,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3f9:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201272.598660:0:14999:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ee9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 6 type: IBT flags: 0x10000000000000 pid: 14999 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.598681:0:14999:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ee9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 6 type: IBT flags: 0x0 pid: 14999 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.598704:0:14999:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ee9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 6 type: IBT flags: 0x0 pid: 14999 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201272.598727:0:14999:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14999:1875064442672000:192.168.201.130@tcp:101:chmod.0 00000100:00100000:2.0:1788201272.598819:0:14999:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bb40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201272.598854:0:15001:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15001:1875064442670848:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201272.599059:0:15001:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.599069:0:15001:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.599076:0:15001:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.599085:0:15001:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.599094:0:15001:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.599101:0:15001:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.599114:0:15001:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201272.599123:0:15001:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c4700 x1875064442670848/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201288 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201272.599133:0:15001:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.599142:0:15001:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 7 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201272.599151:0:15001:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201272.599159:0:15001:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.599166:0:15001:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.599172:0:15001:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.599182:0:15001:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1788201272.599186:0:15001:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201272.599196:0:15001:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:2.0:1788201272.599201:0:15001:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4eaa 00000080:00010000:2.0:1788201272.599205:0:15001:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201272.599207:0:15001:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4eaa from it ffffa0b181d73b58 00010000:00010000:2.0:1788201272.599210:0:15001:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201272.599217:0:15001:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788201272.599231:0:15001:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201272.599235:0:15001:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102c.replay-single, flags=25 00000080:00200000:2.0:1788201272.599239:0:15001:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201272.599247:0:15001:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-6, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), flags=9 00000080:00200000:2.0:1788201272.599250:0:15001:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-6, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), intent=getattr 00000002:00010000:2.0:1788201272.599257:0:15001:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-6,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3f9:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201272.599275:0:15001:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 7 type: IBT flags: 0x10000000000000 pid: 15001 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.599282:0:15001:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 7 type: IBT flags: 0x0 pid: 15001 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.599288:0:15001:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 7 type: IBT flags: 0x0 pid: 15001 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201272.599300:0:15001:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15001:1875064442672256:192.168.201.130@tcp:101:chmod.0 00000100:00100000:2.0:1788201272.599345:0:15001:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bcc0 going to sleep for 11 seconds 00010000:00010000:3.0:1788201272.599573:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201272.599580:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.599593:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351685600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201272.599597:0:13084:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 6 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.599604:0:13084:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 6 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.599612:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 6 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.599625:0:13084:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1788201272.599647:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 6 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.599654:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201272.599683:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 6 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.599690:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c4eb8 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 6 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3e81 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.599703:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201272.599705:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201272.599707:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201272.599716:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201272.599729:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442672384:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201272.600200:0:14997:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689f80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14997:1875064442671360:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201272.600214:0:14997:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 7 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.600227:0:14997:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.600235:0:14997:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.600245:0:14997:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.600255:0:14997:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.600264:0:14997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.600272:0:14997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.600281:0:14997:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201272.600291:0:14997:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352689f80 x1875064442671360/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201288 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201272.600314:0:14997:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201272.600326:0:14997:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.600335:0:14997:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1019 (0 0) 00000080:00200000:3.0:1788201272.600340:0:14997:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f9:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201272.600613:0:14997:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3fb:0x0](ffff9023449aba88): rc = 0 00000080:00200000:3.0:1788201272.600638:0:14997:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3fb:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201272.600650:0:14997:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3fb:0x0](ffff9023449aba88) for lock 0x4605c7c97e1c4ed4 00000080:00010000:3.0:1788201272.600660:0:14997:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3fb:0x0](ffff9023449aba88) 00000080:00010000:3.0:1788201272.600663:0:14997:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4ed4 from it ffffa0b181c8bb20 00010000:00010000:3.0:1788201272.600667:0:14997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.600675:0:14997:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788201272.600705:0:14997:0:(file.c:6283:ll_getattr_dentry()) START file file-2:[0x200000bd1:0x3fb:0x0](ffff9023449aba88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201272.600711:0:14997:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3fb:0x0](ffff9023449aba88),name=file-2 00000002:00010000:3.0:1788201272.600725:0:14997:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3fb:0x0]) in obj [0x200000bd1:0x3f9:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201272.600733:0:14997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201272.600743:0:14997:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201272.600758:0:14997:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f9:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201272.600764:0:14997:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3fb:0x0](ffff9023449aba88) 00000080:00010000:3.0:1788201272.600771:0:14997:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3fb:0x0](ffff9023449aba88) for lock 0x4605c7c97e1c4ed4 00000080:00010000:3.0:1788201272.600779:0:14997:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4ed4 from it ffffa0b181c8bcb8 00010000:00010000:3.0:1788201272.600784:0:14997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201272.600799:0:14997:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201272.600833:0:14997:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3fb:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1788201272.600846:0:14997:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3fb:0x0] 00000008:00010000:3.0:1788201272.600896:0:14997:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788201272.600901:0:14997:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023483ef8b0, osc lock ffff902348866d10, flags 0x400001002 00010000:00010000:3.0:1788201272.600941:0:14997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.600957:0:14997:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201272.601022:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000020:00200000:3.0:1788201272.601046:0:14997:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1788201272.601543:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442672512:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201272.601919:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201272.601926:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c4ecd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3eab expref: -99 pid: 15002 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.601939:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839ca00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201272.601946:0:13086:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c4ecd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3eab expref: -99 pid: 15002 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.605015:0:13086:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c4ecd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d3eab expref: -99 pid: 15002 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.605044:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c4ecd lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d3eab expref: -99 pid: 15002 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201272.605382:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442671872:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201272.605750:0:13086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788201272.605806:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c4ecd lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3eab expref: -99 pid: 15002 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.605814:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201272.605837:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c4ecd lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3eab expref: -99 pid: 15002 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.605846:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c4ecd lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d3eab expref: -99 pid: 15002 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.605862:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201272.605865:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201272.605866:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201272.606024:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201272.606040:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442672896:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201272.609725:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442672384:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201272.619396:0:15000:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b8a80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15000:1875064442671488:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201272.619412:0:15000:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 7 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.619426:0:15000:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.619436:0:15000:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.619445:0:15000:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.619460:0:15000:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.619469:0:15000:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.619477:0:15000:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.619484:0:15000:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201272.619494:0:15000:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458b8a80 x1875064442671488/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201288 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201272.619504:0:15000:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201272.619516:0:15000:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.619524:0:15000:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1022 (0 0) 00000080:00200000:2.0:1788201272.619528:0:15000:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f9:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201272.619578:0:15000:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3fe:0x0](ffff9023449b0908): rc = 0 00000080:00200000:2.0:1788201272.619597:0:15000:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3fe:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201272.619608:0:15000:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3fe:0x0](ffff9023449b0908) for lock 0x4605c7c97e1c4edb 00000080:00010000:2.0:1788201272.619614:0:15000:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3fe:0x0](ffff9023449b0908) 00000080:00010000:2.0:1788201272.619618:0:15000:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4edb from it ffffa0b181d6bb20 00010000:00010000:2.0:1788201272.619621:0:15000:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.619630:0:15000:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788201272.619649:0:15000:0:(file.c:6283:ll_getattr_dentry()) START file file-5:[0x200000bd1:0x3fe:0x0](ffff9023449b0908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201272.619658:0:15000:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3fe:0x0](ffff9023449b0908),name=file-5 00000002:00010000:2.0:1788201272.619667:0:15000:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3fe:0x0]) in obj [0x200000bd1:0x3f9:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201272.619675:0:15000:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201272.619684:0:15000:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201272.619692:0:15000:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f9:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201272.619697:0:15000:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3fe:0x0](ffff9023449b0908) 00000080:00010000:2.0:1788201272.619700:0:15000:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3fe:0x0](ffff9023449b0908) for lock 0x4605c7c97e1c4edb 00000080:00010000:2.0:1788201272.619704:0:15000:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4edb from it ffffa0b181d6bcb8 00010000:00010000:2.0:1788201272.619707:0:15000:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201272.619714:0:15000:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201272.619733:0:15000:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3fe:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201272.619737:0:15000:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3fe:0x0] 00000008:00010000:2.0:1788201272.619756:0:15000:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201272.619758:0:15000:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023533987b0, osc lock ffff9023500dcd10, flags 0x400001002 00010000:00010000:2.0:1788201272.619780:0:15000:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c4f13 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 15000 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.619788:0:15000:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c4f13 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 15000 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201272.619812:0:15000:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1788201272.619839:0:14999:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14999:1875064442672000:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201272.619850:0:14999:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ee9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f45 expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.619860:0:14999:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ee9 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f45 expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.619867:0:14999:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ee9 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f45 expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.619896:0:14999:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ee9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3f45 expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201272.619903:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442673280:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201272.619907:0:14999:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ee9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f45 expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.619919:0:14999:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ee9 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3f45 expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.619927:0:14999:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ee9 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3f45 expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.619935:0:14999:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ee9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3f45 expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201272.619945:0:14999:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c6680 x1875064442672000/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201288 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201272.619955:0:14999:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ee9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f45 expref: -99 pid: 14999 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201272.619966:0:14999:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ee9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f45 expref: -99 pid: 14999 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.619975:0:14999:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1021 (0 0) 00000080:00200000:2.0:1788201272.619980:0:14999:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f9:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201272.620024:0:14999:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3fd:0x0](ffff9023449b63c8): rc = 0 00000080:00200000:2.0:1788201272.620039:0:14999:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3fd:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201272.620047:0:14999:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3fd:0x0](ffff9023449b63c8) for lock 0x4605c7c97e1c4ee9 00000080:00010000:2.0:1788201272.620052:0:14999:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3fd:0x0](ffff9023449b63c8) 00000080:00010000:2.0:1788201272.620055:0:14999:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4ee9 from it ffffa0b181d33b20 00010000:00010000:2.0:1788201272.620059:0:14999:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ee9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3f45 expref: -99 pid: 14999 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.620066:0:14999:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4ee9 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f45 expref: -99 pid: 14999 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788201272.620084:0:14999:0:(file.c:6283:ll_getattr_dentry()) START file file-4:[0x200000bd1:0x3fd:0x0](ffff9023449b63c8), request_mask 2047, flags 0, foreign 0 00000100:00100000:2.0:1788201272.620132:0:15001:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15001:1875064442672256:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201272.620142:0:15001:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.620152:0:15001:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.620158:0:15001:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.620167:0:15001:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.620174:0:15001:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.620183:0:15001:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.620190:0:15001:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.620198:0:15001:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201272.620207:0:15001:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c4700 x1875064442672256/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201288 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201272.620216:0:15001:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201272.620227:0:15001:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.620235:0:15001:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1023 (0 0) 00000080:00200000:2.0:1788201272.620239:0:15001:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f9:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201272.620271:0:15001:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3ff:0x0](ffff9023449a1988): rc = 0 00000080:00200000:2.0:1788201272.620285:0:15001:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3ff:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201272.620294:0:15001:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3ff:0x0](ffff9023449a1988) for lock 0x4605c7c97e1c4ef0 00000080:00010000:2.0:1788201272.620300:0:15001:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3ff:0x0](ffff9023449a1988) 00000080:00010000:2.0:1788201272.620317:0:15001:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4ef0 from it ffffa0b181d73b20 00010000:00010000:2.0:1788201272.620321:0:15001:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.620328:0:15001:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788201272.620347:0:15001:0:(file.c:6283:ll_getattr_dentry()) START file file-6:[0x200000bd1:0x3ff:0x0](ffff9023449a1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201272.620352:0:15001:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3ff:0x0](ffff9023449a1988),name=file-6 00000002:00010000:2.0:1788201272.620361:0:15001:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3ff:0x0]) in obj [0x200000bd1:0x3f9:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201272.620369:0:15001:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201272.620377:0:15001:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201272.620384:0:15001:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f9:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201272.620389:0:15001:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3ff:0x0](ffff9023449a1988) 00000080:00010000:2.0:1788201272.620391:0:15001:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3ff:0x0](ffff9023449a1988) for lock 0x4605c7c97e1c4ef0 00000080:00010000:2.0:1788201272.620395:0:15001:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4ef0 from it ffffa0b181d73cb8 00010000:00010000:2.0:1788201272.620398:0:15001:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201272.620405:0:15001:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201272.620425:0:15001:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3ff:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201272.620430:0:15001:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3ff:0x0] 00000008:00010000:2.0:1788201272.620451:0:15001:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201272.620453:0:15001:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398330, osc lock ffff9023500dd7c0, flags 0x400001002 00010000:00010000:2.0:1788201272.620476:0:15001:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e81829e1/0x4605c7c97e1c4f1a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 15001 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.620485:0:15001:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e81829e1/0x4605c7c97e1c4f1a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 15001 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201272.620508:0:15001:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1788201272.620549:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442673408:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201272.622752:0:15004:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201272.622763:0:15004:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:3.0:1788201272.622770:0:15004:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4edb from it ffffa0b181cdbd78 00010000:00010000:3.0:1788201272.622774:0:15004:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201272.622781:0:15004:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4edb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f14 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201272.622826:0:15004:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201272.622835:0:15004:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:3.0:1788201272.622841:0:15004:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4ef0 from it ffffa0b181cdbd78 00010000:00010000:3.0:1788201272.622845:0:15004:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201272.622852:0:15004:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201272.622869:0:15004:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c4efe lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f4c expref: -99 pid: 15002 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201272.622912:0:15004:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c4efe lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f4c expref: -99 pid: 15002 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:3.0:1788201272.622919:0:15004:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4efe from it ffffa0b181cdbd78 00010000:00010000:3.0:1788201272.622922:0:15004:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c4efe lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3f4c expref: -99 pid: 15002 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201272.622929:0:15004:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c4efe lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3f4c expref: -99 pid: 15002 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201272.622966:0:15004:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ea00/0x4605c7c97e1c4f2f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.622974:0:15004:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ea00/0x4605c7c97e1c4f2f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.622991:0:15004:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f800/0x4605c7c97e1c4f36 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.622996:0:15004:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f800/0x4605c7c97e1c4f36 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.623010:0:15004:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f600/0x4605c7c97e1c4f3d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 6 type: IBT flags: 0x10000000000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.623016:0:15004:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f600/0x4605c7c97e1c4f3d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.623030:0:15004:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ee00/0x4605c7c97e1c4f44 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 7 type: IBT flags: 0x10000000000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.623035:0:15004:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ee00/0x4605c7c97e1c4f44 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x0 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.623048:0:15004:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e800/0x4605c7c97e1c4f4b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 8 type: IBT flags: 0x10000000000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.623054:0:15004:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e800/0x4605c7c97e1c4f4b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x0 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201272.623099:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442673792:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:3.0:1788201272.629299:0:2361:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c4f13 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 15000 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.629343:0:2361:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c4f13 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f84 expref: -99 pid: 15000 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.629358:0:2361:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c4f13 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3f84 expref: -99 pid: 15000 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.629372:0:2361:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c4f13 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f84 expref: -99 pid: 15000 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1788201272.629387:0:2361:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c4f13 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3f84 expref: -99 pid: 15000 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788201272.629399:0:2361:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1788201272.629508:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c4f13 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3f84 expref: -99 pid: 15000 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.629517:0:2361:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c4f13 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3f84 expref: -99 pid: 15000 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.629525:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c4f13 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3f84 expref: -99 pid: 15000 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.630042:0:2361:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c4f13 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3f84 expref: -99 pid: 15000 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1788201272.630073:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442673280:192.168.201.130@tcp:101:chmod.0 00000100:00100000:3.0:1788201272.630097:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442672896:192.168.201.130@tcp:103:ldlm_bl.0 00000080:00200000:3.0:1788201272.632715:0:14996:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x3fa:0x0] updating i_size 0 i_blocks 1 00000020:00010000:3.0:1788201272.632720:0:14996:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235844ee40 (ffff9023500e7758/0) at cl_lock_release():245 00000020:00010000:3.0:1788201272.632723:0:14996:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235844ee40 (ffff9023500e7758/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201272.632728:0:14996:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902345045330 (ffff9023500e7488/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788201272.632733:0:14996:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235844ee40 (ffff9023500e7758/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201272.632735:0:14996:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902345045330 (ffff9023500e7758/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201272.632766:0:14996:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3fa:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201272.632818:0:14996:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201272.632828:0:14996:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201272.632838:0:14996:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788201272.632846:0:14996:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1788201272.632852:0:14996:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201272.632856:0:14996:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201272.632860:0:14996:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4eaa 00000080:00010000:3.0:1788201272.632864:0:14996:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4eaa from it ffffa0b181cf3c08 00010000:00010000:3.0:1788201272.632867:0:14996:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788201272.632899:0:14996:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:3.0:1788201272.632910:0:14996:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201272.632916:0:14996:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102c.replay-single, flags=25 00000080:00200000:3.0:1788201272.632922:0:14996:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:3.0:1788201272.632925:0:14996:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-1, flags=9 00010000:00010000:3.0:1788201272.633002:0:15004:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c4f59 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 8 type: IBT flags: 0x10000000000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.633009:0:15004:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c4f59 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x0 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201272.633062:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201272.633074:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442674560:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:3.0:1788201272.636316:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ea00/0x4605c7c97e1c4f2f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x0 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636328:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ea00/0x4605c7c97e1c4f2f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636333:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ea00/0x4605c7c97e1c4f2f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x10206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636339:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ea00/0x4605c7c97e1c4f2f lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x10206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636347:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ea00/0x4605c7c97e1c4f2f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636354:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351f9ea00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201272.636358:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ea00/0x4605c7c97e1c4f2f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636365:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ea00/0x4605c7c97e1c4f2f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636370:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ea00/0x4605c7c97e1c4f2f lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x20e400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636377:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201272.636383:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ea00/0x4605c7c97e1c4f2f lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x4a0f400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636390:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ea00/0x4605c7c97e1c4f2f lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x4a0f400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636500:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f800/0x4605c7c97e1c4f36 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x0 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636507:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f800/0x4605c7c97e1c4f36 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636512:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f800/0x4605c7c97e1c4f36 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x10206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636516:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f800/0x4605c7c97e1c4f36 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x10206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636522:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f800/0x4605c7c97e1c4f36 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636527:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351f9f800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201272.636687:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f800/0x4605c7c97e1c4f36 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636693:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f800/0x4605c7c97e1c4f36 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636699:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f800/0x4605c7c97e1c4f36 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x20e400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636704:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201272.636707:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f800/0x4605c7c97e1c4f36 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x4a0f400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636713:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f800/0x4605c7c97e1c4f36 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x4a0f400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636727:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f600/0x4605c7c97e1c4f3d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636734:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f600/0x4605c7c97e1c4f3d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636740:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f600/0x4605c7c97e1c4f3d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636745:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f600/0x4605c7c97e1c4f3d lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636750:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f600/0x4605c7c97e1c4f3d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636755:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351f9f600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201272.636758:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f600/0x4605c7c97e1c4f3d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636763:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f600/0x4605c7c97e1c4f3d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636768:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f600/0x4605c7c97e1c4f3d lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x20e400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636773:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201272.636776:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f600/0x4605c7c97e1c4f3d lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636781:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f600/0x4605c7c97e1c4f3d lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636792:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ee00/0x4605c7c97e1c4f44 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636797:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ee00/0x4605c7c97e1c4f44 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636802:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ee00/0x4605c7c97e1c4f44 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636807:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ee00/0x4605c7c97e1c4f44 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636812:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ee00/0x4605c7c97e1c4f44 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636817:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351f9ee00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201272.636819:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ee00/0x4605c7c97e1c4f44 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636824:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ee00/0x4605c7c97e1c4f44 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636829:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ee00/0x4605c7c97e1c4f44 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x20e400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636833:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201272.636836:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ee00/0x4605c7c97e1c4f44 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636840:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9ee00/0x4605c7c97e1c4f44 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636850:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e800/0x4605c7c97e1c4f4b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636855:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e800/0x4605c7c97e1c4f4b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636860:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e800/0x4605c7c97e1c4f4b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636864:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e800/0x4605c7c97e1c4f4b lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.636869:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e800/0x4605c7c97e1c4f4b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.637176:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.637206:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.637220:0:2362:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.637237:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201272.637253:0:2362:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201272.637265:0:2362:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788201272.637275:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.637283:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.637291:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.637298:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201272.637327:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442672512:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201272.637360:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e81829e1/0x4605c7c97e1c4f1a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 15001 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.637375:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e81829e1/0x4605c7c97e1c4f1a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f8b expref: -99 pid: 15001 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.637384:0:2362:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e81829e1/0x4605c7c97e1c4f1a lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3f8b expref: -99 pid: 15001 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.637393:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e81829e1/0x4605c7c97e1c4f1a lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3f8b expref: -99 pid: 15001 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201272.637403:0:2362:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e81829e1/0x4605c7c97e1c4f1a lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d3f8b expref: -99 pid: 15001 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201272.637423:0:2362:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788201272.637431:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e81829e1/0x4605c7c97e1c4f1a lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3f8b expref: -99 pid: 15001 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.637438:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e81829e1/0x4605c7c97e1c4f1a lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3f8b expref: -99 pid: 15001 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.637445:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e81829e1/0x4605c7c97e1c4f1a lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3f8b expref: -99 pid: 15001 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201272.637452:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e81829e1/0x4605c7c97e1c4f1a lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3f8b expref: -99 pid: 15001 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201272.637468:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442673408:192.168.201.130@tcp:101:chmod.0 00000080:00200000:2.0:1788201272.637502:0:15001:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x3ff:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201272.637507:0:15001:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc7530 (ffff902350b9dcf8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201272.637510:0:15001:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc7530 (ffff902350b9dcf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201272.637514:0:15001:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398330 (ffff902350b9d368/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201272.637518:0:15001:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc7530 (ffff902350b9dcf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201272.637520:0:15001:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398330 (ffff902350b9dcf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201272.637766:0:15001:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3ff:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201272.637830:0:15001:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201272.637851:0:15001:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201272.637865:0:15001:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:2.0:1788201272.637910:0:15001:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:2.0:1788201272.637918:0:15001:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201272.637925:0:15001:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201272.637928:0:15001:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4eaa 00000080:00010000:2.0:1788201272.637933:0:15001:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4eaa from it ffffa0b181d73c08 00010000:00010000:2.0:1788201272.637937:0:15001:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:2.0:1788201272.637944:0:15001:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:2.0:1788201272.637961:0:15001:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201272.637967:0:15001:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102c.replay-single, flags=25 00000080:00200000:2.0:1788201272.637972:0:15001:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201272.637976:0:15001:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-6, flags=9 00000080:00200008:2.0:1788201272.637992:0:15001:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-6:[0x200000bd1:0x3ff:0x0](ffff9023449a1988) current size 0, valid attrs 41, mode 8180, uid 1204965376, gid -28637, new size 0, atime 1788201272.000000000, mtime 1788201272.000000000, ctime 1788201272.000000000 00010000:00010000:2.0:1788201272.638010:0:15001:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201272.638023:0:15001:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803600/0x4605c7c97e1c4e79 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3dbd expref: -99 pid: 14934 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201272.638059:0:15001:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201272.638066:0:15001:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201272.638069:0:15001:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c4ef0 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3f53 expref: -99 pid: 15001 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1788201272.638086:0:15001:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 4 is allocated opc 36, max 7 00000100:00100000:2.0:1788201272.638095:0:15001:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c4700 x1875064442675072/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201272.638115:0:15001:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15001:1875064442675072:192.168.201.130@tcp:36:chmod.0 00000100:00100000:2.0:1788201272.638186:0:15001:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bb40 going to sleep for 11 seconds 00010000:00010000:3.0:1788201272.640133:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351f9e800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201272.640142:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e800/0x4605c7c97e1c4f4b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.640152:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e800/0x4605c7c97e1c4f4b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.640160:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e800/0x4605c7c97e1c4f4b lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x20e400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.640168:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201272.640174:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e800/0x4605c7c97e1c4f4b lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x4a0f400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.640182:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e800/0x4605c7c97e1c4f4b lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x4a0f400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201272.640210:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442673792:192.168.201.130@tcp:63:ll_sa.0 00000080:00200000:3.0:1788201272.640254:0:14997:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x3fb:0x0] updating i_size 0 i_blocks 1 00000020:00010000:3.0:1788201272.640260:0:14997:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235000f090 (ffff902346343ab8/0) at cl_lock_release():245 00000020:00010000:3.0:1788201272.640264:0:14997:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235000f090 (ffff902346343ab8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201272.640269:0:14997:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023483ef8b0 (ffff902346343128/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788201272.640273:0:14997:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235000f090 (ffff902346343ab8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201272.640276:0:14997:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023483ef8b0 (ffff902346343ab8/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201272.642728:0:14997:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3fb:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201272.642808:0:14997:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201272.642828:0:14997:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201272.642846:0:14997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1788201272.643051:0:14997:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:3.0:1788201272.643135:0:14997:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201272.643143:0:14997:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201272.643147:0:14997:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4eaa 00000080:00010000:3.0:1788201272.643156:0:14997:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4eaa from it ffffa0b181c8bc08 00010000:00010000:3.0:1788201272.643161:0:14997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1788201272.643170:0:14997:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:3.0:1788201272.643187:0:14997:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201272.643297:0:14997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102c.replay-single, flags=25 00000080:00200000:3.0:1788201272.643313:0:14997:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:3.0:1788201272.643318:0:14997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-2, flags=9 00000080:00200008:3.0:1788201272.643385:0:14997:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-2:[0x200000bd1:0x3fb:0x0](ffff9023449aba88) current size 0, valid attrs 41, mode 8180, uid 1094311936, gid -28637, new size 0, atime 1788201272.000000000, mtime 1788201272.000000000, ctime 1788201272.000000000 00010000:00010000:3.0:1788201272.643417:0:14997:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201272.643436:0:14997:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e400/0x4605c7c97e1c4e41 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d3cf9 expref: -99 pid: 14930 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201272.643480:0:14997:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201272.643492:0:14997:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201272.643495:0:14997:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9f400/0x4605c7c97e1c4ed4 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d3ec7 expref: -99 pid: 14997 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:3.0:1788201272.643516:0:14997:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 6 is allocated opc 36, max 7 00000100:00100000:3.0:1788201272.643543:0:14997:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90235268a300 x1875064442675712/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201272.643570:0:14997:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14997:1875064442675712:192.168.201.130@tcp:36:chmod.0 00000100:00100000:3.0:1788201272.643769:0:14997:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023519380c0 going to sleep for 11 seconds 00010000:00010000:3.0:1788201272.652684:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c4f59 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x0 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.652695:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c4f59 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.652702:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c4f59 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x10206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.652709:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c4f59 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x10206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.652728:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c4f59 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.652738:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db0200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201272.652745:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c4f59 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.652772:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c4f59 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x206400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.652782:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c4f59 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x20e400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.652800:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201272.652807:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c4f59 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x4a0f400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201272.652815:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c4f59 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x4a0f400000000 pid: 15004 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201272.652846:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442674560:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:3.0:1788201272.769036:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201275.843368:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201276.865186:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201279.945118:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201280.961205:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201283.971447:0:14996:0:(client.c:2702:ptlrpc_set_wait()) set ffff902348249f00 going to sleep for 1 seconds 00000100:00100000:3.0:1788201283.973165:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442677248:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201283.973269:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442677504:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201283.973301:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788201283.973308:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442677376:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201283.973332:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442677632:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201283.979599:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442677632:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201283.979624:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442677504:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201284.033289:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201284.994485:0:15002:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0240 going to sleep for 1 seconds 00010000:00010000:3.0:1788201285.057155:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201286.017027:0:15002:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0240 going to sleep for 1 seconds 00000100:00100000:3.0:1788201286.017532:0:14996:0:(client.c:2702:ptlrpc_set_wait()) set ffff902348249f00 going to sleep for 1 seconds 00000100:00100000:3.0:1788201287.041217:0:14996:0:(client.c:2702:ptlrpc_set_wait()) set ffff902348249f00 going to sleep for 1 seconds 00000100:00100000:2.0:1788201287.041378:0:15002:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0240 going to sleep for 1 seconds 00000100:00100000:2.0:1788201287.044067:0:15001:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bb40 going to sleep for 1 seconds 00000100:00100000:2.0:1788201287.047639:0:14997:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023519380c0 going to sleep for 1 seconds 00000100:00100000:3.0:1788201287.048633:0:14998:0:(client.c:2702:ptlrpc_set_wait()) set ffff902348249d80 going to sleep for 1 seconds 00000100:00000400:2.0:1788201288.065088:0:15002:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201272/real 1788201272] req@ffff9023488b0380 x1875064442674176/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201288 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201288.065172:0:15002:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1788201288.065178:0:15002:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff902350a11000: Connection to lustre-MDT0000 (at 192.168.201.130@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00000400:3.0:1788201288.067568:0:14996:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201272/real 1788201272] req@ffff9023458ba680 x1875064442675328/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201288 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201288.172534:0:15002:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00080000:2.0:1788201288.172633:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201288.172638:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201288.172641:0:37:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201288.172651:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 535 00000100:00080000:2.0:1788201288.172664:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788201288.172676:0:37:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201288.172697:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023458c5500 x1875064442677888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201288.172765:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442678016:192.168.201.130@tcp:400:kworker.0 00000100:00080000:3.0:1788201288.173066:0:14996:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 5, was 4): CONNECTING 00000100:00000400:3.0:1788201288.173119:0:14997:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201272/real 1788201272] req@ffff90235268a300 x1875064442675712/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201288 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201288.173138:0:14997:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 5, was 4): CONNECTING 00000100:00100000:3.0:1788201288.173142:0:14997:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023519380c0 going to sleep for 0 seconds 00000100:00000400:3.0:1788201288.173274:0:14998:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201272/real 1788201272] req@ffff9023458bb800 x1875064442676096/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201288 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201288.173290:0:14998:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 5, was 4): CONNECTING 00000100:00100000:3.0:1788201288.173294:0:14998:0:(client.c:2702:ptlrpc_set_wait()) set ffff902348249d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201288.173306:0:14996:0:(client.c:2702:ptlrpc_set_wait()) set ffff902348249f00 going to sleep for 0 seconds 00000100:00100000:3.0:1788201288.173345:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442677888:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201288.174073:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788201288.174083:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442677760:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201288.174124:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442678144:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201288.188566:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442678016:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201288.192794:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442678144:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201289.217304:0:14996:0:(client.c:2702:ptlrpc_set_wait()) set ffff902348249f00 going to sleep for 0 seconds 00010000:00010000:3.0:1788201289.217362:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201289.218846:0:15002:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0240 going to sleep for 0 seconds 00000100:00100000:2.0:1788201290.240982:0:14996:0:(client.c:2702:ptlrpc_set_wait()) set ffff902348249f00 going to sleep for 0 seconds 00010000:00010000:3.0:1788201290.241029:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201290.241281:0:15001:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bb40 going to sleep for 0 seconds 00000100:00100000:3.0:1788201290.243321:0:14998:0:(client.c:2702:ptlrpc_set_wait()) set ffff902348249d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201291.265387:0:14997:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023519380c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788201291.265402:0:15001:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bb40 going to sleep for 0 seconds 00010000:00010000:3.0:1788201291.265475:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201291.265605:0:15000:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe03c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788201291.265620:0:14996:0:(client.c:2702:ptlrpc_set_wait()) set ffff902348249f00 going to sleep for 0 seconds 00000100:00100000:3.0:1788201292.289033:0:14997:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023519380c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788201292.289309:0:15000:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe03c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788201292.289324:0:14996:0:(client.c:2702:ptlrpc_set_wait()) set ffff902348249f00 going to sleep for 0 seconds 00010000:00010000:2.0:1788201292.289615:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201293.314909:0:15000:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe03c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788201293.314983:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201294.283073:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442677248:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201294.283102:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442677760:192.168.201.130@tcp:400:kworker.0 00010000:00010000:3.0:1788201294.337276:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201294.337417:0:15001:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bb40 going to sleep for 0 seconds 00010000:00010000:3.0:1788201295.363528:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201295.369336:0:15001:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bb40 going to sleep for 0 seconds 00010000:00010000:2.0:1788201296.386428:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201296.386530:0:14997:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023519380c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788201297.409535:0:15001:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bb40 going to sleep for 0 seconds 00010000:00010000:2.0:1788201297.410084:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201298.372009:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201288/real 1788201288] req@ffff9023458c5500 x1875064442677888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201298 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201298.372050:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023458c5500 x1875064442677888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201298 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201298.372085:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201298.372114:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788201298.372121:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442677888:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788201298.374993:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442679040:192.168.201.130@tcp:38:kworker.0 00010000:00080000:3.0:1788201298.402168:0:2358:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9023488b1c00 x1875064442679040/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/416 e 0 to 0 dl 1788201313 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201298.402179:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 8589938724 gen 2 00000100:00080000:3.0:1788201298.402249:0:2358:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff902350a11000: connect to target with instance 3 00000100:00080000:3.0:1788201298.402269:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:3.0:1788201298.402274:0:2358:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201298.402276:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 8589938724 gen 2 00000100:00080000:3.0:1788201298.402279:0:2358:0:(recover.c:53:ptlrpc_replay_next()) import ffff902348897000 from lustre-MDT0000_UUID committed 8589938724 last 0 00000100:00080000:3.0:1788201298.402282:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00010000:00080000:3.0:1788201298.405766:0:15224:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff902350a11000 to lustre-MDT0000_UUID@192.168.201.130@tcp 00010000:00010000:3.0:1788201298.405771:0:15224:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff902350a11000 (13) 00000080:00200000:3.0:1788201298.405845:0:15224:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x400:0x0]: losing layout lock 00010000:00010000:3.0:1788201298.405908:0:15224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c4f75 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d4080 expref: -99 pid: 15006 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201298.405932:0:15224:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3ff:0x0]: losing layout lock 00010000:00010000:3.0:1788201298.405945:0:15224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c4f6e lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d4064 expref: -99 pid: 15006 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201298.405965:0:15224:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3fe:0x0]: losing layout lock 00010000:00010000:3.0:1788201298.405977:0:15224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c4f67 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d4048 expref: -99 pid: 15006 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201298.405994:0:15224:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3fd:0x0]: losing layout lock 00010000:00010000:3.0:1788201298.406005:0:15224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c4f60 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d402c expref: -99 pid: 15006 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201298.406023:0:15224:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3fc:0x0]: losing layout lock 00010000:00010000:3.0:1788201298.406033:0:15224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c4e4f lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d3d2a expref: -99 pid: 14931 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201298.406055:0:15224:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3fb:0x0]: losing layout lock 00010000:00010000:3.0:1788201298.406065:0:15224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351f9e400/0x4605c7c97e1c4e41 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d3cf9 expref: -99 pid: 14930 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201298.406086:0:15224:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3fa:0x0]: losing layout lock 00010000:00010000:3.0:1788201298.406096:0:15224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c4e33 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3fa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d3cc8 expref: -99 pid: 14929 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201298.406113:0:15224:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3ff:0x0]: losing layout lock 00010000:00010000:3.0:1788201298.406122:0:15224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803600/0x4605c7c97e1c4e79 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d3dbd expref: -99 pid: 14934 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201298.406140:0:15224:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3fe:0x0]: losing layout lock 00010000:00010000:3.0:1788201298.406148:0:15224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c200/0x4605c7c97e1c4e6b lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d3d8c expref: -99 pid: 14933 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201298.406165:0:15224:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3fd:0x0]: losing layout lock 00010000:00010000:3.0:1788201298.406174:0:15224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c4e5d lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d3d5b expref: -99 pid: 14932 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201298.406190:0:15224:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x400:0x0]: losing layout lock 00010000:00010000:3.0:1788201298.406199:0:15224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c4e87 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x400:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d3dee expref: -99 pid: 14935 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201298.406214:0:15224:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c4e25 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x9383f985bc4d3c97 expref: -99 pid: 14929 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201298.406224:0:15224:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff902350a11000 00010000:00010000:3.0:1788201298.406705:0:15224:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d3e65 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00080000:3.0:1788201298.406734:0:15224:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:2.0:1788201298.446485:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:02000000:2.0:1788201298.504309:0:2358:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff902350a11000: Connection restored to 192.168.201.130@tcp (at 192.168.201.130@tcp) 00000100:00080000:2.0:1788201298.521166:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000109318d6): req@ffff9023458ba680 x1875064442675328/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201288 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201298.521196:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000891fb8d3): req@ffff9023458b9500 x1875064442674432/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201288 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201298.521319:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000035f01d89): req@ffff9023488b0a80 x1875064442674816/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201288 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201298.521335:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000f257b8cb): req@ffff9023488b0380 x1875064442674176/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201288 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201298.521347:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000001aa9a67c): req@ffff90235268a300 x1875064442675712/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201288 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201298.521358:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000533f48cf): req@ffff9023458bb800 x1875064442676096/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201288 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201298.521370:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000001309e2d4): req@ffff9023458c4700 x1875064442675072/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201288 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201298.521382:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442679552:192.168.201.130@tcp:400:ptlrpcd_rcv.0 00000100:00080000:3.0:1788201298.541638:0:14998:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442676096 new x1875064442680448 00000100:00100000:3.0:1788201298.542075:0:14998:0:(client.c:2702:ptlrpc_set_wait()) set ffff902348249d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201298.662190:0:14998:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458bb800 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14998:1875064442676096:192.168.201.130@tcp:36:chmod.0 00000080:00200000:2.0:1788201298.662262:0:14998:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3fc:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:2.0:1788201298.662287:0:14998:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x3fc:0x0](ffff9023449b7448) 00000002:00010000:2.0:1788201298.662298:0:14998:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3fc:0x0]) in obj [0x200000bd1:0x3fc:0x0], intent: layout flags 0 00010000:00010000:2.0:1788201298.662345:0:14998:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4fb4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 14998 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201298.662355:0:14998:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4fb4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 14998 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201298.662365:0:14998:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4fb4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 14998 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201298.662384:0:14998:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14998:1875064442681216:192.168.201.130@tcp:101:chmod.0 00000100:00100000:2.0:1788201298.662437:0:14998:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5ba80 going to sleep for 11 seconds 00000080:00200000:2.0:1788201298.669874:0:15002:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x400:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:2.0:1788201298.669956:0:15002:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x400:0x0](ffff9023449a4b08) 00000002:00010000:2.0:1788201298.669966:0:15002:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x400:0x0]) in obj [0x200000bd1:0x400:0x0], intent: layout flags 0 00010000:00010000:2.0:1788201298.670025:0:15002:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4fbb lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x400:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 15002 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201298.670036:0:15002:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4fbb lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x400:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 15002 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201298.670048:0:15002:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c4fbb lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x400:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 15002 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201298.670084:0:15002:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15002:1875064442681472:192.168.201.130@tcp:101:chmod.0 00000100:00100000:2.0:1788201298.670284:0:15002:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bc00 going to sleep for 11 seconds 00000100:00100000:2.0:1788201298.676232:0:14996:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9180 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14996:1875064442680960:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201298.676281:0:14996:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c4fa6 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x3fa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d43f9 expref: -99 pid: 14996 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201298.676302:0:14996:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c4fa6 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x3fa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d43f9 expref: -99 pid: 14996 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201298.676348:0:14996:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9023458b9180 x1875064442680960/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201314 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201298.676390:0:14996:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c4fa6 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d43f9 expref: -99 pid: 14996 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201298.676410:0:14996:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c4fa6 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d43f9 expref: -99 pid: 14996 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201298.676429:0:14996:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1018 (0 0) 00000080:00200000:2.0:1788201298.676441:0:14996:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x3fa:0x0] suppgids 0 0: rc 0 00000100:00100000:3.0:1788201298.676623:0:14999:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458ba300 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14999:1875064442680704:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201298.676644:0:14999:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4f98 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d43d6 expref: -99 pid: 14999 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201298.676661:0:14999:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4f98 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d43d6 expref: -99 pid: 14999 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201298.676674:0:14999:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9023458ba300 x1875064442680704/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201314 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201298.676684:0:14999:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4f98 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d43d6 expref: -99 pid: 14999 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201298.676695:0:14999:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4f98 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d43d6 expref: -99 pid: 14999 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201298.676705:0:14999:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1021 (0 0) 00000080:00200000:3.0:1788201298.676710:0:14999:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x3fd:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1788201298.676725:0:14999:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3fd:0x0](ffff9023449b63c8) for lock 0x4605c7c97e1c4f98 00000080:00010000:3.0:1788201298.676732:0:14999:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4f98 from it ffffa0b181d33ad8 00010000:00010000:3.0:1788201298.676736:0:14999:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4f98 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d43d6 expref: -99 pid: 14999 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201298.676746:0:14999:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4f98 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d43d6 expref: -99 pid: 14999 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201298.676771:0:14999:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4f98 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d43d6 expref: -99 pid: 14999 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201298.676784:0:14999:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4f98 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d43d6 expref: -99 pid: 14999 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201298.676798:0:14999:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x3fd:0x0](0000000066a1a135) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4f98 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d43d6 expref: -99 pid: 14999 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201298.676926:0:14999:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3fd:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:3.0:1788201298.676962:0:14999:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4f98 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d43d6 expref: -99 pid: 14999 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201298.676971:0:14999:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c4f98 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d43d6 expref: -99 pid: 14999 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:3.0:1788201298.676998:0:14999:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201298.677124:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442681600:192.168.201.130@tcp:2:chmod.0 00000080:00010000:2.0:1788201298.677592:0:14996:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3fa:0x0](ffff9023449b4b08) for lock 0x4605c7c97e1c4fa6 00000080:00010000:2.0:1788201298.677601:0:14996:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4fa6 from it ffffa0b181cf3ad8 00010000:00010000:2.0:1788201298.677606:0:14996:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c4fa6 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d43f9 expref: -99 pid: 14996 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201298.677614:0:14996:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c4fa6 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3fa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d43f9 expref: -99 pid: 14996 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201298.677630:0:14996:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c4fa6 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x3fa:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d43f9 expref: -99 pid: 14996 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201298.677641:0:14996:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c4fa6 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x3fa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d43f9 expref: -99 pid: 14996 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788201298.677649:0:14996:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x3fa:0x0](00000000d67d3de0) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c4fa6 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d43f9 expref: -99 pid: 14996 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201298.677695:0:14996:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3fa:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:2.0:1788201298.677716:0:14996:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c4fa6 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d43f9 expref: -99 pid: 14996 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201298.677725:0:14996:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c4fa6 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3fa:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d43f9 expref: -99 pid: 14996 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1788201298.677739:0:14996:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1788201298.677830:0:15000:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b8e00 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15000:1875064442681088:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201298.677860:0:15000:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803600/0x4605c7c97e1c4fad lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4400 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201298.677906:0:15000:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803600/0x4605c7c97e1c4fad lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4400 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201298.677918:0:15000:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9023458b8e00 x1875064442681088/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201314 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201298.677929:0:15000:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803600/0x4605c7c97e1c4fad lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4400 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201298.677941:0:15000:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803600/0x4605c7c97e1c4fad lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4400 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201298.677949:0:15000:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1022 (0 0) 00000080:00200000:2.0:1788201298.677963:0:15000:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x3fe:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788201298.677976:0:15000:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3fe:0x0](ffff9023449b0908) for lock 0x4605c7c97e1c4fad 00000080:00010000:2.0:1788201298.677981:0:15000:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4fad from it ffffa0b181d6bad8 00010000:00010000:2.0:1788201298.677984:0:15000:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803600/0x4605c7c97e1c4fad lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4400 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201298.677991:0:15000:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803600/0x4605c7c97e1c4fad lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4400 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201298.678003:0:15000:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803600/0x4605c7c97e1c4fad lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4400 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201298.678011:0:15000:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803600/0x4605c7c97e1c4fad lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4400 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788201298.678022:0:15000:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x3fe:0x0](0000000010bb277d) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803600/0x4605c7c97e1c4fad lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4400 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201298.678054:0:15000:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3fe:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:2.0:1788201298.678068:0:15000:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803600/0x4605c7c97e1c4fad lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4400 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201298.678075:0:15000:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803600/0x4605c7c97e1c4fad lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4400 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1788201298.678088:0:15000:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201298.679076:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442681728:192.168.201.130@tcp:2:chmod.0 00000100:00100000:3.0:1788201298.679173:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201298.679181:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442681856:192.168.201.130@tcp:2:chmod.0 00000100:00100000:2.0:1788201298.682322:0:14998:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:14998:1875064442681216:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201298.682361:0:14998:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4fb4 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4407 expref: -99 pid: 14998 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201298.682378:0:14998:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4fb4 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4407 expref: -99 pid: 14998 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201298.682397:0:14998:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9023458c5180 x1875064442681216/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201314 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201298.682409:0:14998:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4fb4 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4407 expref: -99 pid: 14998 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201298.682423:0:14998:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4fb4 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4407 expref: -99 pid: 14998 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201298.682444:0:14998:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1020 (0 0) 00000080:00200000:2.0:1788201298.682454:0:14998:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x3fc:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788201298.684062:0:14998:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3fc:0x0](ffff9023449b7448) for lock 0x4605c7c97e1c4fb4 00000080:00010000:2.0:1788201298.684078:0:14998:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4fb4 from it ffffa0b181d0bad8 00010000:00010000:2.0:1788201298.684084:0:14998:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4fb4 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4407 expref: -99 pid: 14998 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201298.684099:0:14998:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4fb4 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4407 expref: -99 pid: 14998 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201298.684126:0:14998:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4fb4 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4407 expref: -99 pid: 14998 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201298.684138:0:14998:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4fb4 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4407 expref: -99 pid: 14998 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788201298.684147:0:14998:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x3fc:0x0](00000000dcacdbde) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4fb4 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4407 expref: -99 pid: 14998 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201298.684232:0:14998:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3fc:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:2.0:1788201298.684258:0:14998:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4fb4 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4407 expref: -99 pid: 14998 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201298.684267:0:14998:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c4fb4 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4407 expref: -99 pid: 14998 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1788201298.684287:0:14998:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1788201298.684382:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201298.684400:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442682240:192.168.201.130@tcp:2:chmod.0 00000100:00100000:2.0:1788201298.689072:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442681856:192.168.201.130@tcp:2:chmod.0 00000100:00100000:2.0:1788201298.689107:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442681728:192.168.201.130@tcp:2:chmod.0 00000080:00200000:2.0:1788201298.689183:0:14996:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3fa:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201298.689217:0:14996:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-1:[0x200000bd1:0x3fa:0x0](ffff9023449b4b08) current size 0, valid attrs 10000041, mode 8180, uid 1094287360, gid -28637, new size 0, atime 1788201272.000000000, mtime 1788201272.000000000, ctime 1788201272.000000000, rc 0 00000100:00100000:2.0:1788201298.699433:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442681600:192.168.201.130@tcp:2:chmod.0 00000080:00200000:2.0:1788201298.699528:0:14999:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3fd:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201298.699560:0:14999:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-4:[0x200000bd1:0x3fd:0x0](ffff9023449b63c8) current size 0, valid attrs 10000041, mode 8180, uid 1436827648, gid -28637, new size 0, atime 1788201272.000000000, mtime 1788201272.000000000, ctime 1788201272.000000000, rc 0 00000100:00100000:3.0:1788201298.742617:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442682240:192.168.201.130@tcp:2:chmod.0 00000080:00200000:3.0:1788201298.742746:0:15002:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x400:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201298.742779:0:14998:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3fc:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201298.742788:0:15002:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-7:[0x200000bd1:0x400:0x0](ffff9023449a4b08) current size 0, valid attrs 10000041, mode 8180, uid 1117601792, gid -28637, new size 0, atime 1788201272.000000000, mtime 1788201272.000000000, ctime 1788201272.000000000, rc 0 00000080:00200008:2.0:1788201298.742800:0:14998:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-3:[0x200000bd1:0x3fc:0x0](ffff9023449b7448) current size 0, valid attrs 10000041, mode 8180, uid 1436827648, gid -28637, new size 0, atime 1788201272.000000000, mtime 1788201272.000000000, ctime 1788201272.000000000, rc 0 00010000:00010000:2.0:1788201299.457805:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788201299.717550:0:2361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201283/real 1788201283] req@ffff9023458c6300 x1875064442677376/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201299 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201299.717599:0:2361:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9023458c6300 x1875064442677376/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201299 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201299.717626:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442677376:192.168.201.130@tcp:400:kworker.0 00000080:00200000:3.0:1788201302.406306:0:38:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3fa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201302.406331:0:38:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201302.406409:0:38:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3fa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201302.406439:0:38:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3fb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201302.406446:0:38:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201302.406464:0:38:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3fb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201302.406483:0:38:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3fc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201302.406488:0:38:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201302.406505:0:38:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3fc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201302.406522:0:38:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3fd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201302.406527:0:38:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201302.406544:0:38:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3fd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201302.406560:0:38:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3fe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201302.406565:0:38:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201302.406581:0:38:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3fe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201302.406599:0:38:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3ff:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201302.406604:0:38:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201302.406620:0:38:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3ff:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201302.406638:0:38:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x400:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201302.406643:0:38:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201302.406660:0:38:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x400:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1788201302.529128:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201303.425497:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201303.425508:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1788201303.425616:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442682496:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201303.425689:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442682368:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201303.433190:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442682624:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201303.435648:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442682496:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201303.448991:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442682368:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201303.449020:0:2362:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff902350a11000: committing for last_committed 4294968858 gen 2 00000100:00100000:2.0:1788201303.449049:0:2362:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff902352688000 x1875064442681600/t4294968859(4294968859) o2->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 440/432 e 0 to 0 dl 1788201353 ref 1 fl Complete:RQU/604/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201303.449061:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442682624:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201303.558258:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201304.577395:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201304.641053:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201294/real 1788201294] req@ffff9023458bbb80 x1875064442678656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201304.641101:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023458bbb80 x1875064442678656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201304 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201304.641125:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201304.641158:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201304.641163:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442678656:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788201304.644319:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442682752:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788201304.650051:0:2358:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.201.130@tcp: connect to target with instance 0 10000000:01000000:2.0:1788201304.650076:0:2358:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:2.0:1788201304.650087:0:2358:0:(import.c:1284:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x9383f985bc4b09b8 to 0x9383f985bc4d4485 00000100:00080000:2.0:1788201304.666105:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:2.0:1788201304.666129:0:2358:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.201.130@tcp; invalidating 00000100:00100000:2.0:1788201304.667777:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442682752:192.168.201.130@tcp:250:kworker.0 00010000:00010000:3.0:1788201306.673664:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201306.699352:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458ba680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442684160:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000040:00080000:2.0:1788201306.715668:0:15357:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 42 00000020:01000000:2.0:1788201306.716055:0:5601:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 42-41 (rc=0) 10000000:01000000:2.0:1788201306.716081:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:2.0:1788201306.716097:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff902345802a00/0x4605c7c97e1c4fc9 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d4493 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201306.716777:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff902345802a00/0x4605c7c97e1c4fc9 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d4493 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201306.716790:0:5601:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:2.0:1788201306.716793:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log params-ffff902350a11000 from 7 10000000:01000000:2.0:1788201306.716797:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788201306.716858:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff902351685000/0x4605c7c97e1c4fd0 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201306.716868:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff902351685000/0x4605c7c97e1c4fd0 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201306.716919:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff902351685000/0x4605c7c97e1c4fd0 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201306.716954:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442684288:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201306.717070:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bcc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201306.729899:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442684288:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788201306.729943:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff902351685000/0x4605c7c97e1c4fd0 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d449a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201306.729958:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff902351685000/0x4605c7c97e1c4fd0 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d449a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201306.730059:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442684416:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788201306.730145:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023526d76c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201306.735041:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442684416:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788201306.735131:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442684544:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000100:00100000:2.0:1788201306.735240:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b780 going to sleep for 11 seconds 00000100:00100000:2.0:1788201306.758230:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442684544:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000040:00080000:2.0:1788201306.768479:0:15362:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000020:01000000:3.0:1788201306.768714:0:5601:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:3.0:1788201306.768739:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:3.0:1788201306.768757:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff902351685000/0x4605c7c97e1c4fd0 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d449a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201306.768770:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff902351685000/0x4605c7c97e1c4fd0 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d449a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201306.768781:0:5601:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:3.0:1788201306.768785:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-sptlrpc-ffff902350bf19c0 from 1 10000000:01000000:3.0:1788201306.768788:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:3.0:1788201306.768850:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff902342db1200/0x4605c7c97e1c4fd7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201306.768860:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff902342db1200/0x4605c7c97e1c4fd7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201306.768874:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff902342db1200/0x4605c7c97e1c4fd7 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201306.768933:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442684672:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788201306.769047:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023526d76c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201306.773089:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442684672:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788201306.773141:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff902342db1200/0x4605c7c97e1c4fd7 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d44a1 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201306.773157:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff902342db1200/0x4605c7c97e1c4fd7 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d44a1 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201306.773253:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442684800:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788201306.773356:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023526d76c0 going to sleep for 11 seconds 00010000:00010000:2.0:1788201307.718457:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201308.737728:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201310.793697:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201311.812126:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201312.833792:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201314.881171:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201315.908317:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788201316.417270:0:15803:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201316.417313:0:15803:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201316.417338:0:15803:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:3.0:1788201316.417356:0:15803:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:3.0:1788201316.417366:0:15803:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201316.417375:0:15803:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201316.417379:0:15803:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4eaa 00000080:00010000:3.0:1788201316.417387:0:15803:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4eaa from it ffffa0b181c0bb58 00010000:00010000:3.0:1788201316.417392:0:15803:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:3.0:1788201316.417410:0:15803:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:3.0:1788201316.417429:0:15803:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201316.417437:0:15803:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102c.replay-single, flags=24 00000080:00200000:3.0:1788201316.417443:0:15803:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:3.0:1788201316.417455:0:15803:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-2, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), flags=8 00000080:00200000:3.0:1788201316.417460:0:15803:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-2, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), intent=getattr 00000002:00010000:3.0:1788201316.417476:0:15803:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-2,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3f9:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201316.417536:0:15803:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 15803 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.417555:0:15803:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 15803 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.417568:0:15803:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 15803 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201316.417605:0:15803:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15803:1875064442686592:192.168.201.130@tcp:101:checkstat.0 00000100:00100000:3.0:1788201316.417857:0:15803:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005ab40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201316.431508:0:15803:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15803:1875064442686592:192.168.201.130@tcp:101:checkstat.0 00010000:00010000:3.0:1788201316.431550:0:15803:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d44f5 expref: -99 pid: 15803 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.431574:0:15803:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d44f5 expref: -99 pid: 15803 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.431589:0:15803:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d44f5 expref: -99 pid: 15803 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.431603:0:15803:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d44f5 expref: -99 pid: 15803 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.431616:0:15803:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d44f5 expref: -99 pid: 15803 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.431631:0:15803:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d44f5 expref: -99 pid: 15803 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.431640:0:15803:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d44f5 expref: -99 pid: 15803 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.431648:0:15803:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d44f5 expref: -99 pid: 15803 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201316.431661:0:15803:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352688700 x1875064442686592/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201332 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201316.431674:0:15803:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d44f5 expref: -99 pid: 15803 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201316.431695:0:15803:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d44f5 expref: -99 pid: 15803 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.431713:0:15803:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1019 (0 0) 00000080:00200000:3.0:1788201316.431723:0:15803:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f9:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201316.431840:0:15803:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3fb:0x0](ffff9023449aba88): rc = 0 00000080:00200000:3.0:1788201316.431862:0:15803:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3fb:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201316.431873:0:15803:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3fb:0x0](ffff9023449aba88) for lock 0x4605c7c97e1c4fec 00000080:00010000:3.0:1788201316.432352:0:15803:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3fb:0x0](ffff9023449aba88) 00000080:00010000:3.0:1788201316.432357:0:15803:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4fec from it ffffa0b181c0bb20 00010000:00010000:3.0:1788201316.432361:0:15803:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d44f5 expref: -99 pid: 15803 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.432372:0:15803:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d44f5 expref: -99 pid: 15803 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788201316.432424:0:15803:0:(file.c:6283:ll_getattr_dentry()) START file file-2:[0x200000bd1:0x3fb:0x0](ffff9023449aba88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201316.432440:0:15803:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3fb:0x0](ffff9023449aba88),name=file-2 00000002:00010000:3.0:1788201316.432464:0:15803:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3fb:0x0]) in obj [0x200000bd1:0x3f9:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201316.432478:0:15803:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d44f5 expref: -99 pid: 15803 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201316.432491:0:15803:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d44f5 expref: -99 pid: 15803 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201316.432498:0:15803:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f9:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201316.432503:0:15803:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3fb:0x0](ffff9023449aba88) 00000080:00010000:3.0:1788201316.432508:0:15803:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3fb:0x0](ffff9023449aba88) for lock 0x4605c7c97e1c4fec 00000080:00010000:3.0:1788201316.432512:0:15803:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4fec from it ffffa0b181c0bcb8 00010000:00010000:3.0:1788201316.432516:0:15803:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d44f5 expref: -99 pid: 15803 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201316.432523:0:15803:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c4fec lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d44f5 expref: -99 pid: 15803 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201316.432570:0:15803:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3fb:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1788201316.432582:0:15803:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3fb:0x0] 00000008:00010000:3.0:1788201316.432617:0:15803:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788201316.432621:0:15803:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023483ef230, osc lock ffff902348866d10, flags 0x400001002 00010000:00010000:3.0:1788201316.432643:0:15803:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201316.432655:0:15803:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:3.0:1788201316.432671:0:15803:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788201316.432675:0:15803:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201316.432682:0:15803:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:3.0:1788201316.432694:0:15803:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1788201316.432709:0:15803:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x3fb:0x0] updating i_size 0 i_blocks 1 00000020:00010000:3.0:1788201316.432713:0:15803:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235000cde0 (ffff902346343128/0) at cl_lock_release():245 00000020:00010000:3.0:1788201316.432716:0:15803:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235000cde0 (ffff902346343128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201316.432720:0:15803:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023483ef230 (ffff902346343cf8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788201316.432726:0:15803:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235000cde0 (ffff902346343128/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201316.432729:0:15803:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023483ef230 (ffff902346343128/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201316.432754:0:15803:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3fb:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201316.439561:0:15806:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201316.439602:0:15806:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201316.439629:0:15806:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:3.0:1788201316.439652:0:15806:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:3.0:1788201316.439662:0:15806:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201316.439669:0:15806:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201316.439673:0:15806:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4eaa 00000080:00010000:3.0:1788201316.439682:0:15806:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4eaa from it ffffa0b181c13b58 00010000:00010000:3.0:1788201316.439686:0:15806:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:3.0:1788201316.439695:0:15806:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:3.0:1788201316.439716:0:15806:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201316.439725:0:15806:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102c.replay-single, flags=24 00000080:00200000:3.0:1788201316.439732:0:15806:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:3.0:1788201316.439741:0:15806:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-3, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), flags=8 00000080:00200000:3.0:1788201316.439746:0:15806:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-3, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), intent=getattr 00000002:00010000:3.0:1788201316.439762:0:15806:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-3,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3f9:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201316.439823:0:15806:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 15806 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.439833:0:15806:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 15806 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.439845:0:15806:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 15806 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201316.439896:0:15806:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15806:1875064442686720:192.168.201.130@tcp:101:checkstat.0 00000100:00100000:3.0:1788201316.440015:0:15806:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005ab40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201316.447811:0:15806:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15806:1875064442686720:192.168.201.130@tcp:101:checkstat.0 00010000:00010000:3.0:1788201316.447851:0:15806:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4511 expref: -99 pid: 15806 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.447890:0:15806:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4511 expref: -99 pid: 15806 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.447901:0:15806:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4511 expref: -99 pid: 15806 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.447914:0:15806:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4511 expref: -99 pid: 15806 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.447927:0:15806:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4511 expref: -99 pid: 15806 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.448792:0:15806:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4511 expref: -99 pid: 15806 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.448815:0:15806:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4511 expref: -99 pid: 15806 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.448824:0:15806:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4511 expref: -99 pid: 15806 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201316.448841:0:15806:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352688700 x1875064442686720/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201332 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201316.448856:0:15806:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4511 expref: -99 pid: 15806 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201316.448872:0:15806:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4511 expref: -99 pid: 15806 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.448904:0:15806:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1020 (0 0) 00000080:00200000:3.0:1788201316.448913:0:15806:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f9:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201316.452834:0:15806:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3fc:0x0](ffff9023449b7448): rc = 0 00000080:00200000:3.0:1788201316.452893:0:15806:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3fc:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201316.452910:0:15806:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3fc:0x0](ffff9023449b7448) for lock 0x4605c7c97e1c4ff3 00000080:00010000:3.0:1788201316.452927:0:15806:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3fc:0x0](ffff9023449b7448) 00000080:00010000:3.0:1788201316.452931:0:15806:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4ff3 from it ffffa0b181c13b20 00010000:00010000:3.0:1788201316.454258:0:15806:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4511 expref: -99 pid: 15806 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.454283:0:15806:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4511 expref: -99 pid: 15806 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788201316.454345:0:15806:0:(file.c:6283:ll_getattr_dentry()) START file file-3:[0x200000bd1:0x3fc:0x0](ffff9023449b7448), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201316.463407:0:15806:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3fc:0x0](ffff9023449b7448),name=file-3 00000002:00010000:2.0:1788201316.463452:0:15806:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3fc:0x0]) in obj [0x200000bd1:0x3f9:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201316.463478:0:15806:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4511 expref: -99 pid: 15806 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201316.463499:0:15806:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4511 expref: -99 pid: 15806 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201316.463510:0:15806:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f9:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201316.463519:0:15806:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3fc:0x0](ffff9023449b7448) 00000080:00010000:2.0:1788201316.463523:0:15806:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3fc:0x0](ffff9023449b7448) for lock 0x4605c7c97e1c4ff3 00000080:00010000:2.0:1788201316.463532:0:15806:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4ff3 from it ffffa0b181c13cb8 00010000:00010000:2.0:1788201316.463537:0:15806:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4511 expref: -99 pid: 15806 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201316.463560:0:15806:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c4ff3 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4511 expref: -99 pid: 15806 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201316.463630:0:15806:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3fc:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201316.463643:0:15806:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3fc:0x0] 00000008:00010000:2.0:1788201316.463681:0:15806:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201316.463685:0:15806:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398a30, osc lock ffff9023500dd7c0, flags 0x400001002 00010000:00010000:2.0:1788201316.463707:0:15806:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c4f52 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x41e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3fbc expref: -99 pid: 14998 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201316.463721:0:15806:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c4f52 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x41e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3fbc expref: -99 pid: 14998 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788201316.463737:0:15806:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788201316.463741:0:15806:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c4f52 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x41e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3fbc expref: -99 pid: 14998 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201316.463751:0:15806:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c4f52 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x41e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3fbc expref: -99 pid: 14998 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1788201316.463763:0:15806:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788201316.463778:0:15806:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x3fc:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201316.463784:0:15806:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc7530 (ffff902350b9dcf8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201316.463787:0:15806:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc7530 (ffff902350b9dcf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201316.463790:0:15806:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398a30 (ffff902350b9d518/1) at cl_lock_cancel():142 00010000:00010000:3.0:1788201316.464297:0:15808:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ffa lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.464309:0:15808:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ffa lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x0 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.464335:0:15808:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.464341:0:15808:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.464357:0:15808:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.464363:0:15808:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.464377:0:15808:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c500f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 6 type: IBT flags: 0x10000000000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.464383:0:15808:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c500f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.464398:0:15808:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c5016 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 7 type: IBT flags: 0x10000000000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.464404:0:15808:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c5016 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x0 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.464426:0:15808:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c501d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 8 type: IBT flags: 0x10000000000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.464432:0:15808:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c501d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x0 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.464448:0:15808:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c5024 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 9 type: IBT flags: 0x10000000000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.464453:0:15808:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c5024 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x0 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.464469:0:15808:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c502b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 10 type: IBT flags: 0x10000000000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.464475:0:15808:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c502b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x0 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201316.464568:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442686848:192.168.201.130@tcp:63:ll_sa.0 00000020:00010000:2.0:1788201316.464895:0:15806:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc7530 (ffff902350b9dcf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201316.464902:0:15806:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398a30 (ffff902350b9dcf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201316.464944:0:15806:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3fc:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201316.470900:0:15810:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201316.470944:0:15810:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201316.470967:0:15810:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:2.0:1788201316.470986:0:15810:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:2.0:1788201316.470994:0:15810:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201316.471004:0:15810:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201316.471009:0:15810:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4eaa 00000080:00010000:2.0:1788201316.471018:0:15810:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4eaa from it ffffa0b181bfbb58 00010000:00010000:2.0:1788201316.471023:0:15810:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:2.0:1788201316.471033:0:15810:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:2.0:1788201316.471052:0:15810:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201316.471061:0:15810:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102c.replay-single, flags=24 00000080:00200000:2.0:1788201316.471067:0:15810:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201316.471079:0:15810:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-4, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), flags=8 00000080:00200000:2.0:1788201316.471084:0:15810:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-4, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), intent=getattr 00000002:00010000:2.0:1788201316.471122:0:15810:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-4,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3f9:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201316.471182:0:15810:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c5032 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 11 type: IBT flags: 0x10000000000000 pid: 15810 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.471190:0:15810:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c5032 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 11 type: IBT flags: 0x0 pid: 15810 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.471202:0:15810:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c5032 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 11 type: IBT flags: 0x0 pid: 15810 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201316.471235:0:15810:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15810:1875064442686976:192.168.201.130@tcp:101:checkstat.0 00000100:00100000:2.0:1788201316.471525:0:15810:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2cc0 going to sleep for 11 seconds 00010000:00010000:3.0:1788201316.494266:0:2361:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ffa lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d452d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494300:0:2361:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ffa lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d452d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494321:0:2361:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ffa lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d452d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494349:0:2361:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ffa lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d452d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494380:0:2361:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ffa lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d452d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494413:0:2361:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ffa lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d452d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494446:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ffa lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d452d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494457:0:2361:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ffa lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d452d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201316.494473:0:2361:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352688700 x1875064442686848/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3408/7976 e 0 to 0 dl 1788201332 ref 1 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1788201316.494499:0:2361:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ffa lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d452d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1788201316.494564:0:2361:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4ffa from it ffff9023584a5990 00010000:00010000:3.0:1788201316.494577:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ffa lrc: 4/2,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d452d expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494587:0:2361:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ffa lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d452d expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494631:0:2361:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d455e expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494654:0:2361:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d455e expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494660:0:2361:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d455e expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494668:0:2361:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d455e expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494676:0:2361:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d455e expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494684:0:2361:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d455e expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494692:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d455e expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494703:0:2361:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d455e expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201316.494719:0:2361:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352688700 x1875064442686848/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3408/7976 e 0 to 0 dl 1788201332 ref 2 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1788201316.494728:0:2361:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d455e expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1788201316.494741:0:2361:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5001 from it ffff9023584a6990 00010000:00010000:3.0:1788201316.494744:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d455e expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494752:0:2361:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d455e expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201316.494771:0:2361:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494780:0:2361:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494786:0:2361:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494793:0:2361:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494801:0:2361:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494810:0:2361:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494817:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494829:0:2361:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201316.494838:0:2361:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352688700 x1875064442686848/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3408/7976 e 0 to 0 dl 1788201332 ref 3 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1788201316.494847:0:2361:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1788201316.494857:0:2361:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5008 from it ffff9023584a4190 00010000:00010000:3.0:1788201316.494860:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494867:0:2361:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201316.494919:0:2361:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c500f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d459d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494931:0:2361:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c500f lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d459d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494939:0:2361:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c500f lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d459d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494949:0:2361:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c500f lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d459d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494964:0:2361:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c500f lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d459d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494974:0:2361:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c500f lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d459d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494988:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c500f lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d459d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.494996:0:2361:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c500f lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d459d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201316.495010:0:2361:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352688700 x1875064442686848/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3408/7976 e 0 to 0 dl 1788201332 ref 4 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1788201316.495032:0:2361:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c500f lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d459d expref: -99 pid: 15808 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1788201316.495051:0:2361:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c500f from it ffff9023584a6d90 00010000:00010000:3.0:1788201316.495055:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c500f lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d459d expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495063:0:2361:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c500f lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d459d expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201316.495080:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c5016 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x0 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495098:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c5016 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495108:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c5016 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x10206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495114:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c5016 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x10206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495122:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c5016 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495137:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db1800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201316.495143:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c5016 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495159:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c5016 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495178:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c5016 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x20e400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495191:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201316.495199:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c5016 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x4a0f400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495218:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c5016 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x4a0f400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495253:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c501d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495262:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c501d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495268:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c501d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495274:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c501d lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495282:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c501d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495293:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db0600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201316.495296:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c501d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495304:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c501d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495312:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c501d lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x20e400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495318:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201316.495321:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c501d lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495328:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c501d lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495341:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c5024 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495353:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c5024 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495360:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c5024 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495368:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c5024 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495384:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c5024 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495394:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db1400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201316.495400:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c5024 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495409:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c5024 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495419:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c5024 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x20e400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495438:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201316.495445:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c5024 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495463:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c5024 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495477:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c502b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495487:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c502b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495498:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c502b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495514:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c502b lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495528:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c502b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495533:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db1a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201316.495536:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c502b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495555:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c502b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495562:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c502b lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x20e400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495575:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201316.495580:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c502b lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.495586:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c502b lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 15808 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201316.495605:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442686848:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:3.0:1788201316.495644:0:765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ffa lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d452d expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201316.495700:0:765:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3fd:0x0](ffff9023449b63c8): rc = 0 00000080:00200000:3.0:1788201316.495717:0:765:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3fd:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201316.495730:0:765:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3fd:0x0](ffff9023449b63c8) for lock 0x4605c7c97e1c4ffa 00000080:00010000:3.0:1788201316.495735:0:765:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4ffa from it ffff9023584a5990 00010000:00010000:3.0:1788201316.495738:0:765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ffa lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d452d expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201316.495746:0:765:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ffa lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d452d expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201316.495765:0:765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d455e expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201316.495796:0:765:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3fe:0x0](ffff9023449b0908): rc = 0 00000080:00200000:3.0:1788201316.495809:0:765:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3fe:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201316.495818:0:765:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3fe:0x0](ffff9023449b0908) for lock 0x4605c7c97e1c5001 00000080:00010000:3.0:1788201316.495821:0:765:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5001 from it ffff9023584a6990 00010000:00010000:3.0:1788201316.495825:0:765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d455e expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201316.495832:0:765:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d455e expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201316.495848:0:765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201316.495908:0:765:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3ff:0x0](ffff9023449a1988): rc = 0 00000080:00200000:3.0:1788201316.495929:0:765:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3ff:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201316.495940:0:765:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3ff:0x0](ffff9023449a1988) for lock 0x4605c7c97e1c5008 00000080:00010000:3.0:1788201316.495944:0:765:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5008 from it ffff9023584a4190 00010000:00010000:3.0:1788201316.495948:0:765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201316.495957:0:765:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201316.495974:0:765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c500f lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d459d expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201316.496008:0:765:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x400:0x0](ffff9023449a4b08): rc = 0 00000080:00200000:3.0:1788201316.496022:0:765:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x400:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201316.496029:0:765:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x400:0x0](ffff9023449a4b08) for lock 0x4605c7c97e1c500f 00000080:00010000:3.0:1788201316.496033:0:765:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c500f from it ffff9023584a6d90 00010000:00010000:3.0:1788201316.496036:0:765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c500f lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d459d expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201316.496044:0:765:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c500f lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d459d expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201316.498422:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201316.498443:0:5598:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c5032 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d457a expref: -99 pid: 15810 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.498463:0:5598:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351684000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201316.498469:0:5598:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c5032 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d457a expref: -99 pid: 15810 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.498478:0:5598:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c5032 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d457a expref: -99 pid: 15810 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.498494:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c5032 lrc: 3/0,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d457a expref: -99 pid: 15810 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.498525:0:5598:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c4ffa lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d452d expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788201316.498593:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c5032 lrc: 2/0,0 mode: --/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d457a expref: -99 pid: 15810 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.498601:0:5598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201316.498632:0:5598:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c5032 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d457a expref: -99 pid: 15810 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.498640:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c5032 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3fd:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d457a expref: -99 pid: 15810 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.498661:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201316.498663:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201316.498665:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201316.498698:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442687232:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201316.505342:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442687232:192.168.201.130@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1788201316.543158:0:15812:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201316.543189:0:15812:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201316.543208:0:15812:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1788201316.543224:0:15812:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:2.0:1788201316.543231:0:15812:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201316.543238:0:15812:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201316.543255:0:15812:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4eaa 00000080:00010000:2.0:1788201316.543262:0:15812:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4eaa from it ffffa0b181c6bb58 00010000:00010000:2.0:1788201316.543265:0:15812:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1788201316.543273:0:15812:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:2.0:1788201316.543288:0:15812:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201316.543295:0:15812:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102c.replay-single, flags=24 00000080:00200000:2.0:1788201316.543300:0:15812:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201316.543311:0:15812:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-6, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), flags=8 00000080:00200000:2.0:1788201316.543315:0:15812:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-6, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), intent=getattr 00000002:00010000:2.0:1788201316.543331:0:15812:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-6,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3f9:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201316.543376:0:15812:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 15812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.543383:0:15812:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 15812 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.543393:0:15812:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 15812 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201316.543419:0:15812:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4e00 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15812:1875064442687616:192.168.201.130@tcp:101:checkstat.0 00000100:00100000:2.0:1788201316.543523:0:15812:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201316.549586:0:15812:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4e00 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15812:1875064442687616:192.168.201.130@tcp:101:checkstat.0 00010000:00010000:2.0:1788201316.549616:0:15812:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.549635:0:15812:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.549644:0:15812:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.549656:0:15812:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.549668:0:15812:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.549682:0:15812:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.549696:0:15812:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.549703:0:15812:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201316.549716:0:15812:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c4e00 x1875064442687616/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201332 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201316.549728:0:15812:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201316.549745:0:15812:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.549758:0:15812:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201316.549769:0:15812:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201316.549777:0:15812:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.549784:0:15812:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.550352:0:15812:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201316.550379:0:15812:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1788201316.550387:0:15812:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f9:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1788201316.550431:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201316.550445:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.550464:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351684c00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201316.550471:0:13086:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.550481:0:13086:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788201316.550484:0:15812:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x3ff:0x0](ffff9023449a1988): rc = 0 00010000:00010000:3.0:1788201316.550500:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 3/0,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788201316.550513:0:15812:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x3ff:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201316.550527:0:15812:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3ff:0x0](ffff9023449a1988) for lock 0x4605c7c97e1c5008 00010000:00010000:3.0:1788201316.550549:0:13086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1788201316.550551:0:15812:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3ff:0x0](ffff9023449a1988) 00000080:00010000:2.0:1788201316.550565:0:15812:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5008 from it ffffa0b181c6bb20 00010000:00010000:2.0:1788201316.550574:0:15812:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201316.550647:0:15812:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1788201316.550690:0:15812:0:(file.c:6283:ll_getattr_dentry()) START file file-6:[0x200000bd1:0x3ff:0x0](ffff9023449a1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201316.550698:0:15812:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3ff:0x0](ffff9023449a1988),name=file-6 00000002:00010000:2.0:1788201316.550715:0:15812:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3ff:0x0]) in obj [0x200000bd1:0x3f9:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201316.550993:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 2/0,0 mode: --/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.551004:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201316.551038:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.551046:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5047 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d45ff expref: -99 pid: 15812 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.551067:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201316.551069:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201316.551071:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201316.551094:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442687744:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201316.552701:0:15812:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201316.552724:0:15812:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201316.552735:0:15812:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x3f9:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201316.552745:0:15812:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3ff:0x0](ffff9023449a1988) 00000080:00010000:2.0:1788201316.552749:0:15812:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3ff:0x0](ffff9023449a1988) for lock 0x4605c7c97e1c5008 00000080:00010000:2.0:1788201316.552757:0:15812:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5008 from it ffffa0b181c6bcb8 00010000:00010000:2.0:1788201316.552761:0:15812:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201316.552781:0:15812:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5008 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4581 expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201316.553022:0:15812:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x3ff:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201316.553033:0:15812:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x3ff:0x0] 00000008:00010000:2.0:1788201316.553062:0:15812:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201316.553066:0:15812:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398a30, osc lock ffff9023500dd7c0, flags 0x400001002 00010000:00010000:2.0:1788201316.553087:0:15812:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e81829e1/0x4605c7c97e1c4f1a lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x41f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3f8b expref: -99 pid: 15001 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201316.553099:0:15812:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e81829e1/0x4605c7c97e1c4f1a lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3f8b expref: -99 pid: 15001 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788201316.553112:0:15812:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788201316.553116:0:15812:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e81829e1/0x4605c7c97e1c4f1a lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d3f8b expref: -99 pid: 15001 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201316.553124:0:15812:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e81829e1/0x4605c7c97e1c4f1a lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x41f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3f8b expref: -99 pid: 15001 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1788201316.553134:0:15812:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788201316.553146:0:15812:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x3ff:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201316.553150:0:15812:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc7530 (ffff902350b9dcf8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201316.553154:0:15812:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc7530 (ffff902350b9dcf8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201316.553157:0:15812:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398a30 (ffff902350b9d518/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201316.553161:0:15812:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc7530 (ffff902350b9dcf8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201316.553164:0:15812:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398a30 (ffff902350b9dcf8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201316.553191:0:15812:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x3ff:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1788201316.557607:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442687744:192.168.201.130@tcp:103:ldlm_bl.0 00000080:00200000:3.0:1788201316.579227:0:15813:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201316.580399:0:15813:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201316.580422:0:15813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:3.0:1788201316.580438:0:15813:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:3.0:1788201316.580446:0:15813:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201316.580457:0:15813:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201316.580461:0:15813:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4eaa 00000080:00010000:3.0:1788201316.580469:0:15813:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4eaa from it ffffa0b181b53b58 00010000:00010000:3.0:1788201316.580473:0:15813:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:3.0:1788201316.580481:0:15813:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:3.0:1788201316.580497:0:15813:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201316.580507:0:15813:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102c.replay-single, flags=24 00000080:00200000:3.0:1788201316.580514:0:15813:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:3.0:1788201316.580528:0:15813:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-7, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), flags=8 00000080:00200000:3.0:1788201316.580532:0:15813:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-7, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148), intent=getattr 00000002:00010000:3.0:1788201316.580559:0:15813:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-7,[0x0:0x0:0x0]) in obj [0x200000bd1:0x3f9:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201316.580608:0:15813:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c504e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 15813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.580617:0:15813:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c504e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 15813 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.580627:0:15813:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c504e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 15813 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201316.580655:0:15813:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15813:1875064442687872:192.168.201.130@tcp:101:checkstat.0 00000100:00100000:3.0:1788201316.581477:0:15813:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005a0c0 going to sleep for 11 seconds 00010000:00010000:3.0:1788201316.585597:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201316.585621:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c504e lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d461b expref: -99 pid: 15813 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.585635:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db1a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201316.585651:0:13084:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c504e lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d461b expref: -99 pid: 15813 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.585662:0:13084:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c504e lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d461b expref: -99 pid: 15813 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.585674:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c504e lrc: 3/0,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d461b expref: -99 pid: 15813 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.585728:0:13084:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c500f lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d459d expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201316.585798:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c504e lrc: 2/0,0 mode: --/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d461b expref: -99 pid: 15813 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.585825:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201316.585862:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c504e lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d461b expref: -99 pid: 15813 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.585915:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c504e lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x400:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d461b expref: -99 pid: 15813 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.585956:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201316.585960:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201316.585964:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201316.585991:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442688000:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201316.591098:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442688000:192.168.201.130@tcp:103:ldlm_bl.0 00000080:00200000:3.0:1788201316.610814:0:15814:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201316.610856:0:15814:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201316.610909:0:15814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:3.0:1788201316.610932:0:15814:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:3.0:1788201316.610941:0:15814:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201316.611406:0:15814:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201316.611413:0:15814:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4eaa 00000080:00010000:3.0:1788201316.611431:0:15814:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4eaa from it ffffa0b180d4bb58 00010000:00010000:3.0:1788201316.611436:0:15814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:3.0:1788201316.611454:0:15814:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:3.0:1788201316.611476:0:15814:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201316.611487:0:15814:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102c.replay-single, flags=8 00000080:00200008:3.0:1788201316.611501:0:15814:0:(file.c:6283:ll_getattr_dentry()) START file d102c.replay-single:[0x200000bd1:0x3f9:0x0](ffff902344979148), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201316.611507:0:15814:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148),name=d102c.replay-single 00000002:00010000:3.0:1788201316.612419:0:15814:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3f9:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201316.612437:0:15814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c4fde lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d44bd expref: -99 pid: 15798 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201316.612454:0:15814:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c4fde lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d44bd expref: -99 pid: 15798 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201316.612462:0:15814:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201316.612470:0:15814:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x3f9:0x0](ffff902344979148) 00000080:00010000:3.0:1788201316.612473:0:15814:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x3f9:0x0](ffff902344979148) for lock 0x4605c7c97e1c4fde 00000080:00010000:3.0:1788201316.612479:0:15814:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4fde from it ffffa0b180d4bcb8 00010000:00010000:3.0:1788201316.612482:0:15814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c4fde lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d44bd expref: -99 pid: 15798 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201316.612491:0:15814:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c4fde lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d44bd expref: -99 pid: 15798 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201316.612759:0:15814:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201316.612769:0:15814:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201316.612778:0:15814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:3.0:1788201316.612787:0:15814:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:3.0:1788201316.612793:0:15814:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201316.612797:0:15814:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201316.612800:0:15814:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c4eaa 00000080:00010000:3.0:1788201316.612804:0:15814:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c4eaa from it ffffa0b180d4bbf8 00010000:00010000:3.0:1788201316.612932:0:15814:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:3.0:1788201316.612943:0:15814:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:3.0:1788201316.614436:0:15814:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201316.614452:0:15814:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102c.replay-single, flags=266 00000080:00200000:3.0:1788201316.614462:0:15814:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 44 00000080:00200008:3.0:1788201316.614474:0:15814:0:(file.c:1090:ll_file_open()) START file d102c.replay-single:[0x200000bd1:0x3f9:0x0](ffff902344979148), flags 704400 00000002:00010000:3.0:1788201316.614505:0:15814:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x3f9:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:3.0:1788201316.615228:0:15814:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c5055 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 15814 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201316.615238:0:15814:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c5055 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 15814 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201316.615257:0:15814:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201316.615274:0:15814:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352689c00 x1875064442688256/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201316.615288:0:15814:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c5055 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 15814 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201316.616506:0:15814:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15814:1875064442688256:192.168.201.130@tcp:101:rm.0 00000100:00100000:2.0:1788201316.622343:0:15814:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005a840 going to sleep for 11 seconds 00000100:00100000:2.0:1788201316.740117:0:15814:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9500 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15814:1875064442689664:192.168.201.130@tcp:36:rm.0 00000080:00200000:2.0:1788201316.740229:0:15814:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x400:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201316.740244:0:15814:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201316.740306:0:15814:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x400:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201316.740318:0:15814:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x400:0x0](ffff9023449a4b08) 00000008:00010000:2.0:1788201316.740363:0:15814:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 1788201267 1788201267 1788201267 by oinfo size 0 blocks 1 [cma]time 1788201272 1788201267 1788201267 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8ec2fff/0x4605c7c97e1c4f21 lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x420:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d3f92 expref: -99 pid: 15002 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201316.740450:0:15814:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201316.740465:0:15814:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-5, flags=0 00000080:00200000:2.0:1788201316.740472:0:15814:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:2.0:1788201316.740477:0:15814:0:(namei.c:2418:ll_unlink()) VFS Op:name=file-5, dir=[0x200000bd1:0x3f9:0x0](ffff902344979148) 00010000:00010000:2.0:1788201316.740546:0:15814:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d455e expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788201316.740578:0:15814:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3fe:0x0]: losing layout lock 00010000:00010000:2.0:1788201316.740606:0:15814:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803600/0x4605c7c97e1c4fad lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d4400 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201316.740623:0:15814:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x3fe:0x0]: losing layout lock 00010000:00010000:2.0:1788201316.740685:0:15814:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d455e expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788201316.740694:0:15814:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803600/0x4605c7c97e1c4fad lrc: 1/0,0 mode: --/CR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4400 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201316.740700:0:15814:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201316.740704:0:15814:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5001 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d455e expref: -99 pid: 15808 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788201316.740725:0:15814:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803600/0x4605c7c97e1c4fad lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3fe:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4400 expref: -99 pid: 15000 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201316.740750:0:15814:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201316.740758:0:15814:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c4e00 x1875064442689920/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201316.740785:0:15814:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4e00 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15814:1875064442689920:192.168.201.130@tcp:36:rm.0 00000100:00100000:2.0:1788201316.740867:0:15814:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2f00 going to sleep for 11 seconds 00000002:00100000:2.0:1788201316.823329:0:15814:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023458bb480 x1875064442688896/t12884901898(12884901898) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201332 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201316.823343:0:15814:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201316.823350:0:15814:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c5880 x1875064442691072/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201316.823375:0:15814:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15814:1875064442691072:192.168.201.130@tcp:35:rm.0 00000100:00100000:2.0:1788201316.823479:0:15814:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2b40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201316.832606:0:15814:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15814:1875064442691072:192.168.201.130@tcp:35:rm.0 00000002:00100000:2.0:1788201316.832652:0:15814:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023458bb480 x1875064442688896/t12884901898(12884901898) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201332 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1788201316.832756:0:15814:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c5063 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d471e expref: -99 pid: 15814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201316.832767:0:15814:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c505c lrc: 1/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d4637 expref: -99 pid: 15814 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201316.832773:0:15814:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:2.0:1788201316.832778:0:15814:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:2.0:1788201316.832781:0:15814:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c5063 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d471e expref: -99 pid: 15814 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201316.832797:0:15814:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c505c lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x3f9:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d4637 expref: -99 pid: 15814 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201316.832827:0:15814:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c4eaa lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d4239 expref: -99 pid: 14997 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000020:00100000:2.0:1788201316.832850:0:15814:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201316.832857:0:15814:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c5880 x1875064442691328/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201316.832914:0:15814:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15814:1875064442691328:192.168.201.130@tcp:36:rm.0 00000100:00100000:2.0:1788201316.833017:0:15814:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2b40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201316.848350:0:15814:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:15814:1875064442691328:192.168.201.130@tcp:36:rm.0 00000080:00200000:2.0:1788201316.848414:0:15814:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x3f9:0x0](ffff902344979148) 00000100:00100000:3.0:1788201319.186531:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201319.186572:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201319.186612:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458244608 00000100:00100000:3.0:1788201319.186622:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458244608 00000100:00100000:3.0:1788201319.186672:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458244736 00000100:00100000:3.0:1788201319.186675:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458244736 00000100:00100000:3.0:1788201319.186696:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1327 00000100:00100000:3.0:1788201319.186708:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902352688700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10484:x1875064458244608:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201319.186730:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c4f52 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x41e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d3fbc expref: -99 pid: 14998 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201319.186827:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201319.186834:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902352688700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:10484:x1875064458244608:12345-192.168.201.130@tcp:104: Request processed in 128us (316us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201319.186841:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1327 00000100:00100000:3.0:1788201319.186858:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1328 00000100:00100000:3.0:1788201319.186865:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90235268bb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13438:x1875064458244736:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201319.186900:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c4f28 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x41e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d3f99 expref: -99 pid: 14999 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201319.186937:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1788201319.186943:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90235268bb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13438:x1875064458244736:12345-192.168.201.130@tcp:104: Request processed in 79us (371us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201319.186950:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1328 00000100:00100000:3.0:1788201320.322394:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442691712:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201320.322508:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442691968:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201320.322538:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788201320.322543:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442691840:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201320.322562:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442692096:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201320.328519:0:2361:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 12884901907 gen 2 00000100:00100000:3.0:1788201320.328565:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442691840:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201320.328581:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442692096:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201320.328590:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442691968:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201320.334847:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442691712:192.168.201.130@tcp:400:kworker.0 00010000:00010000:3.0:1788201321.050233:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201324.272055:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201324.272094:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201324.272134:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201324.273614:0:5591:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1788201324.273644:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458246528 00000100:00100000:3.0:1788201324.273655:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458246528 00000100:00100000:3.0:1788201324.273903:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1329 00000100:00100000:3.0:1788201324.273915:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023451b9050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13446:x1875064458246528:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201324.273932:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201324.274010:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201324.274017:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023451b9050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13446:x1875064458246528:12345-192.168.201.130@tcp:104: Request processed in 105us (1972us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201324.274024:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1329 00010000:00010000:3.0:1788201324.274183:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201324.274191:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201324.274201:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351f9e000 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201324.274206:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201324.274295:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201324.274302:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201324.274323:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201324.274330:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dd21cd3a/0x4605c7c97e1c4ef7 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3f5a expref: -99 pid: 14997 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201324.274346:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201324.274348:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201324.274350:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201324.274366:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442692224:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201324.292986:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201324.293120:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201324.293214:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458246912 00000100:00100000:2.0:1788201324.293226:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458246912 00000100:00100000:2.0:1788201324.293272:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458247040 00000100:00100000:2.0:1788201324.293276:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458247040 00000100:00100000:2.0:1788201324.293293:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1330 00000100:00100000:2.0:1788201324.293303:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c7100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:13449:x1875064458246912:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201324.293325:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c4f0c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x41d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d3f7d expref: -99 pid: 14996 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201324.293655:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201324.293664:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c7100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:13449:x1875064458246912:12345-192.168.201.130@tcp:104: Request processed in 362us (685us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201324.293671:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1330 00000100:00100000:2.0:1788201324.293690:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1331 00000100:00100000:2.0:1788201324.293697:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023458c4000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13444:x1875064458247040:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201324.293707:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8ec2fff/0x4605c7c97e1c4f21 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x420:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d3f92 expref: -99 pid: 15002 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201324.293748:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:2.0:1788201324.293754:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023458c4000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13444:x1875064458247040:12345-192.168.201.130@tcp:104: Request processed in 57us (635us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201324.293761:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1331 00010000:00010000:1.0:1788201324.295270:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8ec2fff/0x4605c7c97e1c4f21 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x420:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3f92 expref: -99 pid: 15002 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201324.295282:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201324.295288:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8ec2fff/0x4605c7c97e1c4f21 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x420:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3f92 expref: -99 pid: 15002 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201324.295294:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8ec2fff/0x4605c7c97e1c4f21 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x420:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d3f92 expref: -99 pid: 15002 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201324.295306:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201324.295308:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201324.295310:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201324.295331:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442692608:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201324.297624:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201324.297644:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442692736:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201324.297935:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201324.298457:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442692224:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201324.323110:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442692736:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201324.323127:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442692608:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201324.323138:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442692480:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201324.323147:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442692352:192.168.201.130@tcp:103:ldlm_bl.0 00000001:02000400:1.0:1788201324.660578:0:16371:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 102d: check replay 00000080:00200008:3.0:1788201325.050448:0:16387:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788201325.050465:0:16387:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201325.050487:0:16387:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201325.050516:0:16387:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c506a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4772 expref: -99 pid: 16387 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201325.050532:0:16387:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c506a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4772 expref: -99 pid: 16387 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788201325.050540:0:16387:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201325.050548:0:16387:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201325.050551:0:16387:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c506a 00000080:00010000:3.0:1788201325.050558:0:16387:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c506a from it ffffa0b1816ebcb8 00010000:00010000:3.0:1788201325.050562:0:16387:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c506a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4772 expref: -99 pid: 16387 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201325.050570:0:16387:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c506a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4772 expref: -99 pid: 16387 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201325.050742:0:16387:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201325.050751:0:16387:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201325.050758:0:16387:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c506a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4772 expref: -99 pid: 16387 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201325.050767:0:16387:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c506a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4772 expref: -99 pid: 16387 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201325.050772:0:16387:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201325.050777:0:16387:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201325.050780:0:16387:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c506a 00000080:00010000:3.0:1788201325.050784:0:16387:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c506a from it ffffa0b1816ebbb0 00010000:00010000:3.0:1788201325.050787:0:16387:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c506a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4772 expref: -99 pid: 16387 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201325.050863:0:16387:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c506a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4772 expref: -99 pid: 16387 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788201325.050873:0:16387:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 44 00000080:00200008:3.0:1788201325.050909:0:16387:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 300000 00000080:00200080:3.0:1788201325.050928:0:16387:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c03066f0 arg=7ffefd51b8f0 00000080:00200000:3.0:1788201325.050942:0:16387:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) name=d102d.replay-single stripe_offset=-1 stripe_count=1, hash_type=0 00010000:00010000:3.0:1788201325.050969:0:16387:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c506a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d4772 expref: -99 pid: 16387 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788201325.051223:0:16387:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c506a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d4772 expref: -99 pid: 16387 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788201325.051231:0:16387:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201325.051234:0:16387:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c506a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d4772 expref: -99 pid: 16387 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000020:00100000:3.0:1788201325.051256:0:16387:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201325.051267:0:16387:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352688700 x1875064442693248/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 568/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201325.051290:0:16387:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16387:1875064442693248:192.168.201.130@tcp:36:lfs.0 00000100:00100000:3.0:1788201325.051381:0:16387:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201325.081462:0:16387:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16387:1875064442693248:192.168.201.130@tcp:36:lfs.0 00000080:00200000:1.0:1788201325.081547:0:16387:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988) 00000080:00200000:1.0:1788201325.081556:0:16387:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x402:0x0], updating i_size 512 00000080:00200008:1.0:1788201325.081616:0:16387:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 300000 00000080:00200000:2.0:1788201325.124467:0:16390:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201325.124495:0:16390:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201325.124555:0:16390:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 16390 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.124566:0:16390:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16390 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.124576:0:16390:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16390 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201325.124602:0:16390:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16390:1875064442693376:192.168.201.130@tcp:101:lfs.0 00000100:00100000:2.0:1788201325.124699:0:16390:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358f5a480 going to sleep for 11 seconds 00000002:00100000:0.0:1788201325.143158:0:16390:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b1880 x1875064442693632/t12884901909(12884901909) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/608 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201325.143229:0:16390:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 44 00000080:00200000:0.0:1788201325.143281:0:16390:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201325.143296:0:16390:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201325.143311:0:16390:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201325.143323:0:16390:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201325.143330:0:16390:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201325.143335:0:16390:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201325.143339:0:16390:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:0.0:1788201325.143343:0:16390:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181bdbbf8 00010000:00010000:0.0:1788201325.143347:0:16390:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201325.143356:0:16390:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201325.143371:0:16390:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201325.143385:0:16390:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=d102d.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), file ffff902346275980, open_flags 28800, mode 0 opened 0 00000080:00200000:0.0:1788201325.143400:0:16390:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d102d.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=open 00000002:00010000:0.0:1788201325.143415:0:16390:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d102d.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0504001 00010000:00010000:0.0:1788201325.143465:0:16390:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c5086 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 16390 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.143475:0:16390:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c5086 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16390 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201325.143486:0:16390:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201325.143624:0:16390:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b1180 x1875064442693888/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201325.143634:0:16390:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c5086 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16390 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201325.143725:0:16390:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1180 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16390:1875064442693888:192.168.201.130@tcp:101:lfs.0 00000100:00100000:1.0:1788201325.143937:0:16390:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201325.154064:0:16390:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1180 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16390:1875064442693888:192.168.201.130@tcp:101:lfs.0 00010000:00010000:1.0:1788201325.154095:0:16390:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c5086 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47db expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.154118:0:16390:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c5086 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47db expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.154126:0:16390:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c5086 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47db expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.154137:0:16390:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c5086 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47db expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.154147:0:16390:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c5086 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47db expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.154161:0:16390:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c5086 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47db expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.154169:0:16390:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c5086 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47db expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.154176:0:16390:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c5086 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47db expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201325.154193:0:16390:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9023488b1180 x1875064442693888/t12884901910(12884901910) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/4688 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201325.154207:0:16390:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b1180 x1875064442693888/t12884901910(12884901910) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/4688 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201325.154217:0:16390:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c5086 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47db expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.154228:0:16390:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1026 (0 0) 00000080:00200000:1.0:1788201325.154234:0:16390:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201325.154257:0:16390:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x402:0x0], updating i_size 512 00000080:00200000:1.0:1788201325.154260:0:16390:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x402:0x0](ffff9023449a1988): rc = 0 00000080:00010000:1.0:1788201325.154265:0:16390:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x402:0x0](ffff9023449a1988) for lock 0x4605c7c97e1c5086 00000080:00010000:1.0:1788201325.154344:0:16390:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x402:0x0](ffff9023449a1988) 00000080:00200008:1.0:1788201325.154360:0:16390:0:(file.c:1090:ll_file_open()) START file d102d.replay-single:[0x200000bd1:0x402:0x0](ffff9023449a1988), flags 504000 00000002:00100000:1.0:1788201325.154380:0:16390:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b1180 x1875064442693888/t12884901910(12884901910) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/4688 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201325.154432:0:16390:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b1180 x1875064442693888/t12884901910(12884901910) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/4688 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201325.154442:0:16390:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201325.154446:0:16390:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458baa00 x1875064442694144/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201325.154468:0:16390:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458baa00 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16390:1875064442694144:192.168.201.130@tcp:35:lfs.0 00000100:00100000:1.0:1788201325.154563:0:16390:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201325.157559:0:16390:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458baa00 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16390:1875064442694144:192.168.201.130@tcp:35:lfs.0 00000002:00100000:1.0:1788201325.157598:0:16390:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b1180 x1875064442693888/t12884901910(12884901910) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/4688 e 0 to 0 dl 1788201341 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201325.157685:0:16390:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5086 from it ffff902353388b40 00010000:00010000:1.0:1788201325.157695:0:16390:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c5086 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47db expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.157708:0:16390:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c5086 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47db expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201325.157731:0:16390:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 44 00000080:00200008:1.0:1788201325.157785:0:16390:0:(file.c:479:ll_file_release()) START file d102d.replay-single:[0x200000bd1:0x402:0x0](ffff9023449a1988), flags 304000 00000080:00200080:1.0:1788201325.157828:0:16390:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988) cmd=c03066f1 arg=1abb3e0 00000100:00100000:1.0:1788201325.157942:0:16390:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458baa00 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16390:1875064442694272:192.168.201.130@tcp:33:lfs.0 00000100:00100000:1.0:1788201325.159474:0:16390:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201325.177055:0:16390:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458baa00 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16390:1875064442694272:192.168.201.130@tcp:33:lfs.0 00000080:00200080:1.0:1788201325.177108:0:16390:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988) cmd=800466af arg=7fff7f1228e0 00000080:00200008:1.0:1788201325.177259:0:16390:0:(file.c:479:ll_file_release()) START file d102d.replay-single:[0x200000bd1:0x402:0x0](ffff9023449a1988), flags 504000 00000002:00100000:1.0:1788201325.177318:0:16390:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b1880 x1875064442693632/t12884901909(12884901909) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/608 e 0 to 0 dl 1788201341 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201325.177331:0:16390:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201325.177336:0:16390:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458baa00 x1875064442694528/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201325.177358:0:16390:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458baa00 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16390:1875064442694528:192.168.201.130@tcp:35:lfs.0 00000100:00100000:1.0:1788201325.177467:0:16390:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa3c0 going to sleep for 11 seconds 00010000:00010000:0.0:1788201325.185153:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201325.186286:0:16390:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458baa00 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16390:1875064442694528:192.168.201.130@tcp:35:lfs.0 00000002:00100000:1.0:1788201325.186321:0:16390:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b1880 x1875064442693632/t12884901909(12884901909) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/608 e 0 to 0 dl 1788201341 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201325.294820:0:16393:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201325.294850:0:16393:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201325.294870:0:16393:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201325.294916:0:16393:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201325.294924:0:16393:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201325.294933:0:16393:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201325.294937:0:16393:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:3.0:1788201325.294944:0:16393:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181bebbf8 00010000:00010000:3.0:1788201325.294949:0:16393:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201325.294959:0:16393:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201325.294976:0:16393:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201325.294988:0:16393:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d102d.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=25 00000080:00200000:3.0:1788201325.294993:0:16393:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d102d.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=0 00000002:00010000:3.0:1788201325.295005:0:16393:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d102d.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1788201325.295052:0:16393:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c508d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 16393 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.295060:0:16393:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c508d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16393 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.295074:0:16393:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c508d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16393 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201325.295100:0:16393:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16393:1875064442694656:192.168.201.130@tcp:101:touch.0 00000100:00100000:3.0:1788201325.295201:0:16393:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0240 going to sleep for 11 seconds 00000100:00100000:3.0:1788201325.301261:0:16393:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16393:1875064442694656:192.168.201.130@tcp:101:touch.0 00010000:00010000:3.0:1788201325.301289:0:16393:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c508d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47fe expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.301305:0:16393:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c508d lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47fe expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.301313:0:16393:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c508d lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47fe expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.301323:0:16393:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c508d lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47fe expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.301333:0:16393:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c508d lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47fe expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.301346:0:16393:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c508d lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47fe expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.301430:0:16393:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c508d lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47fe expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.301441:0:16393:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c508d lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47fe expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201325.301454:0:16393:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff90235268bb80 x1875064442694656/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201341 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201325.301467:0:16393:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c508d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47fe expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.301476:0:16393:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1026 (0 0) 00000080:00200000:3.0:1788201325.301483:0:16393:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201325.301505:0:16393:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x402:0x0], updating i_size 512 00000080:00200000:3.0:1788201325.301508:0:16393:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x402:0x0](ffff9023449a1988): rc = 0 00000080:00010000:3.0:1788201325.301513:0:16393:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x402:0x0](ffff9023449a1988) for lock 0x4605c7c97e1c508d 00000080:00010000:3.0:1788201325.301524:0:16393:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x402:0x0](ffff9023449a1988) 00000080:00010000:3.0:1788201325.301528:0:16393:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c508d from it ffffa0b181bebac8 00010000:00010000:3.0:1788201325.301531:0:16393:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c508d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47fe expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.301538:0:16393:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c508d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47fe expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201325.301568:0:16393:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201325.301587:0:16393:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201325.301593:0:16393:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file-1, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), file ffff902352cc8900, open_flags 8941, mode 81b6 opened 0 00000080:00200000:3.0:1788201325.301600:0:16393:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-1, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=open|creat 00000002:00010000:3.0:1788201325.301622:0:16393:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-1,[0x200000bd1:0x403:0x0]) in obj [0x200000bd1:0x402:0x0], intent: open|creat flags 0104502 00010000:00010000:3.0:1788201325.301631:0:16393:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c5086 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d47db expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201325.301707:0:16393:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c5086 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d47db expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201325.304044:0:16393:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201325.304050:0:16393:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c5086 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d47db expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201325.304099:0:16393:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 16393 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.304107:0:16393:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16393 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201325.304120:0:16393:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201325.304134:0:16393:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90235268bb80 x1875064442694912/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201325.304145:0:16393:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16393 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201325.304171:0:16393:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16393:1875064442694912:192.168.201.130@tcp:101:touch.0 00000100:00100000:3.0:1788201325.304411:0:16393:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0240 going to sleep for 11 seconds 00000100:00100000:3.0:1788201325.333308:0:16393:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16393:1875064442694912:192.168.201.130@tcp:101:touch.0 00010000:00010000:3.0:1788201325.333336:0:16393:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d481a expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.333363:0:16393:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d481a expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.333373:0:16393:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d481a expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.333383:0:16393:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d481a expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.333408:0:16393:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d481a expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.333422:0:16393:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d481a expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.333430:0:16393:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d481a expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.333438:0:16393:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d481a expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201325.333450:0:16393:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff90235268bb80 x1875064442694912/t12884901913(12884901913) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201325.333464:0:16393:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90235268bb80 x1875064442694912/t12884901913(12884901913) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201325.333479:0:16393:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d481a expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201325.333491:0:16393:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d481a expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.333501:0:16393:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1027 (0 0) 00000080:00200000:3.0:1788201325.333508:0:16393:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201325.333522:0:16393:0:(namei.c:1628:ll_create_it()) VFS Op:name=file-1, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=open|creat 00000080:00200000:3.0:1788201325.333553:0:16393:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x403:0x0](ffff90234497ec08) 00000080:00200000:3.0:1788201325.333672:0:16393:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x403:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201325.333682:0:16393:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x403:0x0](ffff90234497ec08) 00000080:00010000:3.0:1788201325.333685:0:16393:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x403:0x0](ffff90234497ec08) for lock 0x4605c7c97e1c5094 00000080:00010000:3.0:1788201325.333693:0:16393:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x403:0x0](ffff90234497ec08) 00000080:00200008:3.0:1788201325.333705:0:16393:0:(file.c:1090:ll_file_open()) START file file-1:[0x200000bd1:0x403:0x0](ffff90234497ec08), flags 104501 00000002:00100000:3.0:1788201325.333735:0:16393:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90235268bb80 x1875064442694912/t12884901913(12884901913) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201325.333774:0:16393:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5094 from it ffff9023533ea900 00010000:00010000:3.0:1788201325.333779:0:16393:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d481a expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.333789:0:16393:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d481a expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201325.333810:0:16393:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x403:0x0](ffff90234497ec08), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201325.333844:0:16393:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-1:[0x200000bd1:0x403:0x0](ffff90234497ec08) current size 0, valid attrs 20070, mode 9023, uid -2118205872, gid -24399, new size 18446744071973566999, atime 1788201325.000000000, mtime 1788201325.000000000, ctime 1788201325.000000000 00010000:00010000:3.0:1788201325.333861:0:16393:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d481a expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201325.333907:0:16393:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x403:0x0]: losing layout lock 00010000:00010000:3.0:1788201325.333946:0:16393:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d481a expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201325.333952:0:16393:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201325.333954:0:16393:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c5094 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d481a expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788201325.333974:0:16393:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201325.333981:0:16393:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90235268ad80 x1875064442695168/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201325.334000:0:16393:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16393:1875064442695168:192.168.201.130@tcp:36:touch.0 00000100:00100000:3.0:1788201325.334073:0:16393:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0b40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201325.341514:0:16393:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16393:1875064442695168:192.168.201.130@tcp:36:touch.0 00000080:00200000:3.0:1788201325.341592:0:16393:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x403:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:3.0:1788201325.341611:0:16393:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x403:0x0](ffff90234497ec08) 00000002:00010000:3.0:1788201325.341621:0:16393:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x403:0x0]) in obj [0x200000bd1:0x403:0x0], intent: layout flags 0 00010000:00010000:3.0:1788201325.341667:0:16393:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c509b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 16393 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.341675:0:16393:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c509b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16393 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.341685:0:16393:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c509b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16393 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201325.341702:0:16393:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16393:1875064442695296:192.168.201.130@tcp:101:touch.0 00000100:00100000:3.0:1788201325.341775:0:16393:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0b40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201325.345113:0:16393:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16393:1875064442695296:192.168.201.130@tcp:101:touch.0 00010000:00010000:3.0:1788201325.345143:0:16393:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c509b lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d482f expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.345158:0:16393:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c509b lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d482f expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201325.345174:0:16393:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff90235268a300 x1875064442695296/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201341 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201325.345184:0:16393:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c509b lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d482f expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201325.345196:0:16393:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c509b lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d482f expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.345207:0:16393:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1027 (0 0) 00000080:00200000:3.0:1788201325.345213:0:16393:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x403:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1788201325.345230:0:16393:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x403:0x0](ffff90234497ec08) for lock 0x4605c7c97e1c509b 00000080:00010000:3.0:1788201325.345238:0:16393:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c509b from it ffffa0b181beba78 00010000:00010000:3.0:1788201325.345242:0:16393:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c509b lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d482f expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.345252:0:16393:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c509b lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d482f expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201325.345274:0:16393:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c509b lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d482f expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201325.345287:0:16393:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c509b lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d482f expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201325.345295:0:16393:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x403:0x0](000000003e1b269b) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c509b lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d482f expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201325.345358:0:16393:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x403:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:3.0:1788201325.345378:0:16393:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c509b lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d482f expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201325.345400:0:16393:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c509b lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d482f expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:3.0:1788201325.345422:0:16393:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201325.345520:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201325.345536:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442695424:192.168.201.130@tcp:2:touch.0 00000100:00100000:3.0:1788201325.357581:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442695424:192.168.201.130@tcp:2:touch.0 00000080:00200000:0.0:1788201325.357757:0:16393:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x403:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788201325.357779:0:16393:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-1:[0x200000bd1:0x403:0x0](ffff90234497ec08) current size 0, valid attrs 100201f0, mode 9023, uid -2118205872, gid -24399, new size 18446744071973566999, atime 1788201325.000000000, mtime 1788201325.000000000, ctime 1788201325.000000000, rc 0 00000080:00200008:0.0:1788201325.357814:0:16393:0:(file.c:479:ll_file_release()) START file file-1:[0x200000bd1:0x403:0x0](ffff90234497ec08), flags 104001 00000002:00100000:0.0:1788201325.357916:0:16393:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff90235268bb80 x1875064442694912/t12884901913(12884901913) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201325.357926:0:16393:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201325.357932:0:16393:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b0700 x1875064442695680/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201325.357952:0:16393:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b0700 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16393:1875064442695680:192.168.201.130@tcp:35:touch.0 00000100:00100000:0.0:1788201325.358034:0:16393:0:(client.c:2702:ptlrpc_set_wait()) set ffff902345676c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201325.361604:0:16393:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b0700 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16393:1875064442695680:192.168.201.130@tcp:35:touch.0 00000002:00100000:0.0:1788201325.361639:0:16393:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff90235268bb80 x1875064442694912/t12884901913(12884901913) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201325.386231:0:16394:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201325.386277:0:16394:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201325.386298:0:16394:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201325.386314:0:16394:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201325.386322:0:16394:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201325.386330:0:16394:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201325.386334:0:16394:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:1.0:1788201325.386341:0:16394:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181bfbbf8 00010000:00010000:1.0:1788201325.386345:0:16394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201325.386354:0:16394:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201325.386370:0:16394:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201325.386378:0:16394:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:1.0:1788201325.386396:0:16394:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201325.386409:0:16394:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201325.386414:0:16394:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file-2, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), file ffff9023558cb980, open_flags 8941, mode 81b6 opened 0 00000080:00200000:1.0:1788201325.386424:0:16394:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-2, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=open|creat 00000002:00010000:1.0:1788201325.386445:0:16394:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-2,[0x200000bd1:0x404:0x0]) in obj [0x200000bd1:0x402:0x0], intent: open|creat flags 0104502 00010000:00010000:1.0:1788201325.386505:0:16394:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 16394 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.386512:0:16394:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16394 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201325.386525:0:16394:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201325.386537:0:16394:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458baa00 x1875064442695936/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201325.386546:0:16394:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16394 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201325.386568:0:16394:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458baa00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16394:1875064442695936:192.168.201.130@tcp:101:touch.0 00000100:00100000:1.0:1788201325.386717:0:16394:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201325.394215:0:16394:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458baa00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16394:1875064442695936:192.168.201.130@tcp:101:touch.0 00010000:00010000:1.0:1788201325.394256:0:16394:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d484b expref: -99 pid: 16394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.394279:0:16394:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d484b expref: -99 pid: 16394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.394289:0:16394:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d484b expref: -99 pid: 16394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.394301:0:16394:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d484b expref: -99 pid: 16394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.394313:0:16394:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d484b expref: -99 pid: 16394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.394327:0:16394:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d484b expref: -99 pid: 16394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.394336:0:16394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d484b expref: -99 pid: 16394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.394343:0:16394:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d484b expref: -99 pid: 16394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201325.394354:0:16394:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023458baa00 x1875064442695936/t12884901916(12884901916) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201325.394367:0:16394:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458baa00 x1875064442695936/t12884901916(12884901916) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201325.394379:0:16394:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d484b expref: -99 pid: 16394 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201325.394405:0:16394:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d484b expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.394415:0:16394:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1028 (0 0) 00000080:00200000:1.0:1788201325.394421:0:16394:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201325.394434:0:16394:0:(namei.c:1628:ll_create_it()) VFS Op:name=file-2, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=open|creat 00000080:00200000:1.0:1788201325.394462:0:16394:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x404:0x0](ffff9023449a7448) 00000080:00200000:1.0:1788201325.394587:0:16394:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x404:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201325.394598:0:16394:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x404:0x0](ffff9023449a7448) 00000080:00010000:1.0:1788201325.394602:0:16394:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x404:0x0](ffff9023449a7448) for lock 0x4605c7c97e1c50a2 00000080:00010000:1.0:1788201325.394612:0:16394:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x404:0x0](ffff9023449a7448) 00000080:00200008:1.0:1788201325.394625:0:16394:0:(file.c:1090:ll_file_open()) START file file-2:[0x200000bd1:0x404:0x0](ffff9023449a7448), flags 104501 00000002:00100000:1.0:1788201325.394643:0:16394:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458baa00 x1875064442695936/t12884901916(12884901916) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201325.394681:0:16394:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c50a2 from it ffff9023450e7a20 00010000:00010000:1.0:1788201325.394686:0:16394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d484b expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.394695:0:16394:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d484b expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201325.394716:0:16394:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x404:0x0](ffff9023449a7448), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201325.394758:0:16394:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-2:[0x200000bd1:0x404:0x0](ffff9023449a7448) current size 0, valid attrs 20070, mode 9023, uid -2118140336, gid -24399, new size 18446744071973566999, atime 1788201325.000000000, mtime 1788201325.000000000, ctime 1788201325.000000000 00010000:00010000:1.0:1788201325.394780:0:16394:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d484b expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201325.394805:0:16394:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x404:0x0]: losing layout lock 00010000:00010000:1.0:1788201325.394851:0:16394:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d484b expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201325.394858:0:16394:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201325.394861:0:16394:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50a2 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d484b expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201325.394911:0:16394:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201325.394919:0:16394:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458b8e00 x1875064442696192/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201325.394941:0:16394:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b8e00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16394:1875064442696192:192.168.201.130@tcp:36:touch.0 00000100:00100000:1.0:1788201325.395006:0:16394:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fab40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201325.402219:0:16394:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b8e00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16394:1875064442696192:192.168.201.130@tcp:36:touch.0 00000080:00200000:1.0:1788201325.402310:0:16394:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x404:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:1.0:1788201325.402334:0:16394:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x404:0x0](ffff9023449a7448) 00000002:00010000:1.0:1788201325.402345:0:16394:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x404:0x0]) in obj [0x200000bd1:0x404:0x0], intent: layout flags 0 00010000:00010000:1.0:1788201325.402414:0:16394:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c50a9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 16394 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.402425:0:16394:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c50a9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16394 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.402437:0:16394:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c50a9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16394 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201325.402458:0:16394:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9180 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16394:1875064442696320:192.168.201.130@tcp:101:touch.0 00000100:00100000:1.0:1788201325.402605:0:16394:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fab40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201325.408515:0:16394:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9180 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16394:1875064442696320:192.168.201.130@tcp:101:touch.0 00010000:00010000:1.0:1788201325.408547:0:16394:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c50a9 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4860 expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.408563:0:16394:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c50a9 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4860 expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201325.408577:0:16394:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9023458b9180 x1875064442696320/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201341 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201325.408588:0:16394:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c50a9 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4860 expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201325.408599:0:16394:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c50a9 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4860 expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.408611:0:16394:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1028 (0 0) 00000080:00200000:1.0:1788201325.408617:0:16394:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x404:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201325.408634:0:16394:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x404:0x0](ffff9023449a7448) for lock 0x4605c7c97e1c50a9 00000080:00010000:1.0:1788201325.408642:0:16394:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c50a9 from it ffffa0b181bfba78 00010000:00010000:1.0:1788201325.408646:0:16394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c50a9 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4860 expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.408656:0:16394:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c50a9 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4860 expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201325.408677:0:16394:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c50a9 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4860 expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201325.408687:0:16394:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c50a9 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4860 expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201325.408694:0:16394:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x404:0x0](00000000f3d86d2f) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c50a9 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4860 expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201325.408770:0:16394:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x404:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:1.0:1788201325.408794:0:16394:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c50a9 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4860 expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201325.408802:0:16394:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c50a9 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4860 expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1788201325.408821:0:16394:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:1.0:1788201325.408936:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201325.408951:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442696448:192.168.201.130@tcp:2:touch.0 00000100:00100000:1.0:1788201325.417680:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442696448:192.168.201.130@tcp:2:touch.0 00000080:00200000:0.0:1788201325.417865:0:16394:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x404:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1788201325.417928:0:16394:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-2:[0x200000bd1:0x404:0x0](ffff9023449a7448) current size 0, valid attrs 100201f0, mode 9023, uid -2118140336, gid -24399, new size 18446744071973566999, atime 1788201325.000000000, mtime 1788201325.000000000, ctime 1788201325.000000000, rc 0 00000080:00200008:0.0:1788201325.417957:0:16394:0:(file.c:479:ll_file_release()) START file file-2:[0x200000bd1:0x404:0x0](ffff9023449a7448), flags 104001 00000002:00100000:0.0:1788201325.418019:0:16394:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023458baa00 x1875064442695936/t12884901916(12884901916) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201325.418031:0:16394:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201325.418036:0:16394:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b2a00 x1875064442696704/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201325.418058:0:16394:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b2a00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16394:1875064442696704:192.168.201.130@tcp:35:touch.0 00000100:00100000:0.0:1788201325.418139:0:16394:0:(client.c:2702:ptlrpc_set_wait()) set ffff902345676c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201325.426132:0:16394:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b2a00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16394:1875064442696704:192.168.201.130@tcp:35:touch.0 00000002:00100000:0.0:1788201325.426181:0:16394:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023458baa00 x1875064442695936/t12884901916(12884901916) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201325.430737:0:16395:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201325.430764:0:16395:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201325.430785:0:16395:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201325.430807:0:16395:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201325.430817:0:16395:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201325.430824:0:16395:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201325.430828:0:16395:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:0.0:1788201325.430836:0:16395:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181bcbbf8 00010000:00010000:0.0:1788201325.430840:0:16395:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201325.430849:0:16395:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788201325.430866:0:16395:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201325.430874:0:16395:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:0.0:1788201325.430923:0:16395:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201325.430935:0:16395:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201325.430939:0:16395:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file-3, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), file ffff902346274600, open_flags 8941, mode 81b6 opened 0 00000080:00200000:0.0:1788201325.430947:0:16395:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-3, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=open|creat 00000002:00010000:0.0:1788201325.430966:0:16395:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-3,[0x200000bd1:0x405:0x0]) in obj [0x200000bd1:0x402:0x0], intent: open|creat flags 0104502 00010000:00010000:0.0:1788201325.431030:0:16395:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 16395 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.431038:0:16395:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16395 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201325.431051:0:16395:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201325.431065:0:16395:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b1500 x1875064442696960/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201325.431074:0:16395:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16395 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201325.431100:0:16395:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1500 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16395:1875064442696960:192.168.201.130@tcp:101:touch.0 00000100:00100000:0.0:1788201325.435620:0:16395:0:(client.c:2702:ptlrpc_set_wait()) set ffff902345676c00 going to sleep for 11 seconds 00000100:00100000:2.0:1788201325.441330:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442697216:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201325.441347:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442697088:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201325.445788:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442697088:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201325.445800:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442697216:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201325.448566:0:16395:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1500 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16395:1875064442696960:192.168.201.130@tcp:101:touch.0 00010000:00010000:0.0:1788201325.448594:0:16395:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d487c expref: -99 pid: 16395 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.448617:0:16395:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d487c expref: -99 pid: 16395 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.448625:0:16395:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d487c expref: -99 pid: 16395 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.448634:0:16395:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d487c expref: -99 pid: 16395 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.448646:0:16395:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d487c expref: -99 pid: 16395 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.448659:0:16395:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d487c expref: -99 pid: 16395 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.448667:0:16395:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d487c expref: -99 pid: 16395 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.448673:0:16395:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d487c expref: -99 pid: 16395 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201325.448684:0:16395:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023488b1500 x1875064442696960/t12884901919(12884901919) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201325.448696:0:16395:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b1500 x1875064442696960/t12884901919(12884901919) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201325.448714:0:16395:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d487c expref: -99 pid: 16395 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201325.448730:0:16395:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d487c expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.448743:0:16395:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1029 (0 0) 00000080:00200000:0.0:1788201325.448752:0:16395:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201325.448765:0:16395:0:(namei.c:1628:ll_create_it()) VFS Op:name=file-3, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=open|creat 00000080:00200000:0.0:1788201325.448793:0:16395:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x405:0x0](ffff9023449b5348) 00000080:00200000:1.0:1788201325.456522:0:16395:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x405:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201325.456554:0:16395:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x405:0x0](ffff9023449b5348) 00000080:00010000:1.0:1788201325.456558:0:16395:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x405:0x0](ffff9023449b5348) for lock 0x4605c7c97e1c50b0 00000080:00010000:1.0:1788201325.456573:0:16395:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x405:0x0](ffff9023449b5348) 00000080:00200008:1.0:1788201325.456587:0:16395:0:(file.c:1090:ll_file_open()) START file file-3:[0x200000bd1:0x405:0x0](ffff9023449b5348), flags 104501 00000002:00100000:1.0:1788201325.456620:0:16395:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b1500 x1875064442696960/t12884901919(12884901919) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201325.456656:0:16395:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c50b0 from it ffff902353388720 00010000:00010000:1.0:1788201325.456663:0:16395:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d487c expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.456677:0:16395:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d487c expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201325.456772:0:16395:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x405:0x0](ffff9023449b5348), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201325.456811:0:16395:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-3:[0x200000bd1:0x405:0x0](ffff9023449b5348) current size 0, valid attrs 20070, mode 9023, uid -2118336944, gid -24399, new size 18446744071973566999, atime 1788201325.000000000, mtime 1788201325.000000000, ctime 1788201325.000000000 00010000:00010000:1.0:1788201325.456834:0:16395:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d487c expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201325.456862:0:16395:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x405:0x0]: losing layout lock 00010000:00010000:1.0:1788201325.456928:0:16395:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d487c expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201325.456935:0:16395:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201325.456938:0:16395:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c50b0 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d487c expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788201325.456961:0:16395:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201325.456968:0:16395:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458ba300 x1875064442697472/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201325.456993:0:16395:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458ba300 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16395:1875064442697472:192.168.201.130@tcp:36:touch.0 00000100:00100000:1.0:1788201325.457085:0:16395:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fab40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201325.467040:0:16395:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458ba300 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16395:1875064442697472:192.168.201.130@tcp:36:touch.0 00000080:00200000:1.0:1788201325.467183:0:16395:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x405:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:1.0:1788201325.467204:0:16395:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x405:0x0](ffff9023449b5348) 00000002:00010000:1.0:1788201325.467215:0:16395:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x405:0x0]) in obj [0x200000bd1:0x405:0x0], intent: layout flags 0 00010000:00010000:1.0:1788201325.467267:0:16395:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50b7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 16395 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.467281:0:16395:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50b7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16395 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.467291:0:16395:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50b7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16395 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201325.467310:0:16395:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458bb100 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16395:1875064442697600:192.168.201.130@tcp:101:touch.0 00000100:00100000:1.0:1788201325.467390:0:16395:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fab40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201325.469622:0:16395:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458bb100 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16395:1875064442697600:192.168.201.130@tcp:101:touch.0 00010000:00010000:1.0:1788201325.469653:0:16395:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50b7 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4891 expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.469668:0:16395:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50b7 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4891 expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201325.469683:0:16395:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9023458bb100 x1875064442697600/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201341 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201325.469693:0:16395:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50b7 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4891 expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201325.469705:0:16395:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50b7 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4891 expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.469731:0:16395:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1029 (0 0) 00000080:00200000:1.0:1788201325.469737:0:16395:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x405:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201325.469753:0:16395:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x405:0x0](ffff9023449b5348) for lock 0x4605c7c97e1c50b7 00000080:00010000:1.0:1788201325.469762:0:16395:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c50b7 from it ffffa0b181bcba78 00010000:00010000:1.0:1788201325.469766:0:16395:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50b7 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4891 expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201325.469776:0:16395:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50b7 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4891 expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201325.469798:0:16395:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50b7 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4891 expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201325.469809:0:16395:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50b7 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4891 expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201325.469818:0:16395:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x405:0x0](00000000484282c9) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50b7 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4891 expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201325.469908:0:16395:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x405:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:1.0:1788201325.469932:0:16395:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50b7 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4891 expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201325.469941:0:16395:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50b7 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4891 expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1788201325.469969:0:16395:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1788201325.470165:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442697728:192.168.201.130@tcp:2:touch.0 00000100:00100000:0.0:1788201325.473658:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442697728:192.168.201.130@tcp:2:touch.0 00000080:00200000:1.0:1788201325.474859:0:16395:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x405:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201325.474926:0:16395:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-3:[0x200000bd1:0x405:0x0](ffff9023449b5348) current size 0, valid attrs 100201f0, mode 9023, uid -2118336944, gid -24399, new size 18446744071973566999, atime 1788201325.000000000, mtime 1788201325.000000000, ctime 1788201325.000000000, rc 0 00000080:00200008:1.0:1788201325.474957:0:16395:0:(file.c:479:ll_file_release()) START file file-3:[0x200000bd1:0x405:0x0](ffff9023449b5348), flags 104001 00000002:00100000:1.0:1788201325.475022:0:16395:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b1500 x1875064442696960/t12884901919(12884901919) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201325.475035:0:16395:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201325.475040:0:16395:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458b8000 x1875064442697984/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201325.475063:0:16395:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b8000 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16395:1875064442697984:192.168.201.130@tcp:35:touch.0 00000100:00100000:1.0:1788201325.475146:0:16395:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fab40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201325.481400:0:16395:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b8000 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16395:1875064442697984:192.168.201.130@tcp:35:touch.0 00000002:00100000:1.0:1788201325.481438:0:16395:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b1500 x1875064442696960/t12884901919(12884901919) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201325.505954:0:16396:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201325.505980:0:16396:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201325.506000:0:16396:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201325.506016:0:16396:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788201325.506024:0:16396:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201325.506032:0:16396:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201325.506035:0:16396:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:2.0:1788201325.506043:0:16396:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181dc7bf8 00010000:00010000:2.0:1788201325.506047:0:16396:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201325.506056:0:16396:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788201325.506072:0:16396:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201325.506079:0:16396:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:2.0:1788201325.506085:0:16396:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201325.506096:0:16396:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201325.506101:0:16396:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file-4, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), file ffff9023454c0780, open_flags 8941, mode 81b6 opened 0 00000080:00200000:2.0:1788201325.506109:0:16396:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-4, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=open|creat 00000002:00010000:2.0:1788201325.506130:0:16396:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-4,[0x200000bd1:0x406:0x0]) in obj [0x200000bd1:0x402:0x0], intent: open|creat flags 0104502 00010000:00010000:2.0:1788201325.506182:0:16396:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 16396 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.506192:0:16396:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16396 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201325.506206:0:16396:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201325.506218:0:16396:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c4380 x1875064442698240/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201325.506232:0:16396:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16396 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201325.506253:0:16396:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4380 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16396:1875064442698240:192.168.201.130@tcp:101:touch.0 00000100:00100000:2.0:1788201325.514699:0:16396:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358f5a480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201325.521756:0:16396:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4380 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16396:1875064442698240:192.168.201.130@tcp:101:touch.0 00010000:00010000:3.0:1788201325.521783:0:16396:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48ad expref: -99 pid: 16396 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.521809:0:16396:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48ad expref: -99 pid: 16396 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.521819:0:16396:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48ad expref: -99 pid: 16396 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.521831:0:16396:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d48ad expref: -99 pid: 16396 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.521842:0:16396:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48ad expref: -99 pid: 16396 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.521855:0:16396:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d48ad expref: -99 pid: 16396 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.521864:0:16396:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d48ad expref: -99 pid: 16396 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.521872:0:16396:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d48ad expref: -99 pid: 16396 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201325.522009:0:16396:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023458c4380 x1875064442698240/t12884901922(12884901922) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201325.522023:0:16396:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458c4380 x1875064442698240/t12884901922(12884901922) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201325.522038:0:16396:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48ad expref: -99 pid: 16396 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201325.522051:0:16396:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48ad expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.522063:0:16396:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1030 (0 0) 00000080:00200000:3.0:1788201325.522069:0:16396:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201325.522084:0:16396:0:(namei.c:1628:ll_create_it()) VFS Op:name=file-4, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=open|creat 00000080:00200000:3.0:1788201325.522121:0:16396:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x406:0x0](ffff902344979148) 00000080:00200000:3.0:1788201325.522230:0:16396:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x406:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201325.522239:0:16396:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x406:0x0](ffff902344979148) 00000080:00010000:3.0:1788201325.522242:0:16396:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x406:0x0](ffff902344979148) for lock 0x4605c7c97e1c50be 00000080:00010000:3.0:1788201325.522251:0:16396:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x406:0x0](ffff902344979148) 00000080:00200008:3.0:1788201325.522689:0:16396:0:(file.c:1090:ll_file_open()) START file file-4:[0x200000bd1:0x406:0x0](ffff902344979148), flags 104501 00000002:00100000:3.0:1788201325.522806:0:16396:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458c4380 x1875064442698240/t12884901922(12884901922) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201325.522846:0:16396:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c50be from it ffff902342d49b40 00010000:00010000:3.0:1788201325.522852:0:16396:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d48ad expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.522861:0:16396:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d48ad expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201325.523435:0:16396:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x406:0x0](ffff902344979148), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201325.523475:0:16396:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-4:[0x200000bd1:0x406:0x0](ffff902344979148) current size 0, valid attrs 20070, mode 9023, uid -2116256176, gid -24399, new size 18446744071973566999, atime 1788201325.000000000, mtime 1788201325.000000000, ctime 1788201325.000000000 00010000:00010000:3.0:1788201325.523622:0:16396:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d48ad expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201325.523663:0:16396:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x406:0x0]: losing layout lock 00010000:00010000:3.0:1788201325.524819:0:16396:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d48ad expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201325.524831:0:16396:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201325.524835:0:16396:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684600/0x4605c7c97e1c50be lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d48ad expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788201325.524865:0:16396:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201325.525086:0:16396:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90235268aa00 x1875064442698496/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201325.525993:0:16396:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16396:1875064442698496:192.168.201.130@tcp:36:touch.0 00000100:00100000:2.0:1788201325.526105:0:16396:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358f5a480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201325.536473:0:16396:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16396:1875064442698496:192.168.201.130@tcp:36:touch.0 00000080:00200000:2.0:1788201325.536567:0:16396:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x406:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:2.0:1788201325.536589:0:16396:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x406:0x0](ffff902344979148) 00000002:00010000:2.0:1788201325.536601:0:16396:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x406:0x0]) in obj [0x200000bd1:0x406:0x0], intent: layout flags 0 00010000:00010000:2.0:1788201325.536669:0:16396:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c50c5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 16396 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.536679:0:16396:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c50c5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16396 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.536690:0:16396:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c50c5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16396 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201325.536711:0:16396:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5c00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16396:1875064442698624:192.168.201.130@tcp:101:touch.0 00000100:00100000:2.0:1788201325.536847:0:16396:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358f5a480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201325.551608:0:16396:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5c00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16396:1875064442698624:192.168.201.130@tcp:101:touch.0 00010000:00010000:3.0:1788201325.551638:0:16396:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c50c5 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d48c2 expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.551653:0:16396:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c50c5 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48c2 expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201325.551667:0:16396:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9023458c5c00 x1875064442698624/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201341 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201325.551679:0:16396:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c50c5 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48c2 expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201325.551693:0:16396:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c50c5 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48c2 expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.551704:0:16396:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1030 (0 0) 00000080:00200000:3.0:1788201325.551711:0:16396:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x406:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1788201325.551725:0:16396:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x406:0x0](ffff902344979148) for lock 0x4605c7c97e1c50c5 00000080:00010000:3.0:1788201325.551733:0:16396:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c50c5 from it ffffa0b181dc7a78 00010000:00010000:3.0:1788201325.551737:0:16396:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c50c5 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d48c2 expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.551746:0:16396:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c50c5 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48c2 expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201325.551766:0:16396:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c50c5 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48c2 expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201325.551777:0:16396:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c50c5 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48c2 expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201325.551784:0:16396:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x406:0x0](000000005cda43cf) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c50c5 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48c2 expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201325.551843:0:16396:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x406:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:3.0:1788201325.551860:0:16396:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c50c5 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d48c2 expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201325.551868:0:16396:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c50c5 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d48c2 expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1788201325.555724:0:16396:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201325.555911:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442698752:192.168.201.130@tcp:2:touch.0 00000100:00100000:3.0:1788201325.566933:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442698752:192.168.201.130@tcp:2:touch.0 00000080:00200000:2.0:1788201325.572618:0:16396:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x406:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201325.572820:0:16396:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-4:[0x200000bd1:0x406:0x0](ffff902344979148) current size 0, valid attrs 100201f0, mode 9023, uid -2116256176, gid -24399, new size 18446744071973566999, atime 1788201325.000000000, mtime 1788201325.000000000, ctime 1788201325.000000000, rc 0 00000080:00200008:2.0:1788201325.572857:0:16396:0:(file.c:479:ll_file_release()) START file file-4:[0x200000bd1:0x406:0x0](ffff902344979148), flags 104001 00000002:00100000:2.0:1788201325.572943:0:16396:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023458c4380 x1875064442698240/t12884901922(12884901922) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201325.572956:0:16396:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201325.572962:0:16396:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c7b80 x1875064442699008/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201325.572990:0:16396:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16396:1875064442699008:192.168.201.130@tcp:35:touch.0 00000100:00100000:2.0:1788201325.573140:0:16396:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358f5a480 going to sleep for 11 seconds 00000100:00100000:0.0:1788201325.587830:0:16396:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7b80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16396:1875064442699008:192.168.201.130@tcp:35:touch.0 00000002:00100000:0.0:1788201325.587870:0:16396:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023458c4380 x1875064442698240/t12884901922(12884901922) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201325.597647:0:16397:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201325.597676:0:16397:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201325.597695:0:16397:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201325.597712:0:16397:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1788201325.597727:0:16397:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201325.597735:0:16397:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201325.597739:0:16397:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:0.0:1788201325.597745:0:16397:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181cdbbf8 00010000:00010000:0.0:1788201325.597750:0:16397:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1788201325.597759:0:16397:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:0.0:1788201325.597776:0:16397:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201325.597783:0:16397:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:0.0:1788201325.597789:0:16397:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201325.597800:0:16397:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201325.597805:0:16397:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file-5, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), file ffff902343d62180, open_flags 8941, mode 81b6 opened 0 00000080:00200000:0.0:1788201325.597813:0:16397:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-5, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=open|creat 00000002:00010000:0.0:1788201325.597833:0:16397:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-5,[0x200000bd1:0x407:0x0]) in obj [0x200000bd1:0x402:0x0], intent: open|creat flags 0104502 00010000:00010000:0.0:1788201325.597920:0:16397:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 16397 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.597929:0:16397:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16397 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201325.597942:0:16397:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201325.597954:0:16397:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b2d80 x1875064442699264/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201325.597969:0:16397:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16397 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201325.597990:0:16397:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b2d80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16397:1875064442699264:192.168.201.130@tcp:101:touch.0 00000100:00100000:0.0:1788201325.598112:0:16397:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023456766c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201325.610044:0:16397:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b2d80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16397:1875064442699264:192.168.201.130@tcp:101:touch.0 00010000:00010000:0.0:1788201325.610072:0:16397:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48de expref: -99 pid: 16397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.610096:0:16397:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48de expref: -99 pid: 16397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.610103:0:16397:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48de expref: -99 pid: 16397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.610115:0:16397:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d48de expref: -99 pid: 16397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.610125:0:16397:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48de expref: -99 pid: 16397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.610138:0:16397:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d48de expref: -99 pid: 16397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.610146:0:16397:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d48de expref: -99 pid: 16397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.610159:0:16397:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d48de expref: -99 pid: 16397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201325.610170:0:16397:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023488b2d80 x1875064442699264/t12884901925(12884901925) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201325.610182:0:16397:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b2d80 x1875064442699264/t12884901925(12884901925) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201325.610195:0:16397:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48de expref: -99 pid: 16397 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201325.610207:0:16397:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48de expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.610217:0:16397:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1031 (0 0) 00000080:00200000:0.0:1788201325.610223:0:16397:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201325.610247:0:16397:0:(namei.c:1628:ll_create_it()) VFS Op:name=file-5, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=open|creat 00000080:00200000:0.0:1788201325.610281:0:16397:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x407:0x0](ffff9023449b4b08) 00000080:00200000:0.0:1788201325.610403:0:16397:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x407:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201325.610414:0:16397:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x407:0x0](ffff9023449b4b08) 00000080:00010000:0.0:1788201325.610417:0:16397:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x407:0x0](ffff9023449b4b08) for lock 0x4605c7c97e1c50cc 00000080:00010000:0.0:1788201325.610427:0:16397:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x407:0x0](ffff9023449b4b08) 00000080:00200008:0.0:1788201325.610441:0:16397:0:(file.c:1090:ll_file_open()) START file file-5:[0x200000bd1:0x407:0x0](ffff9023449b4b08), flags 104501 00000002:00100000:0.0:1788201325.610459:0:16397:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b2d80 x1875064442699264/t12884901925(12884901925) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201325.610700:0:16397:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c50cc from it ffff902353388f00 00010000:00010000:0.0:1788201325.610706:0:16397:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d48de expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.610780:0:16397:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d48de expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201325.610806:0:16397:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x407:0x0](ffff9023449b4b08), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201325.610842:0:16397:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-5:[0x200000bd1:0x407:0x0](ffff9023449b4b08) current size 0, valid attrs 20070, mode 9023, uid -2117222832, gid -24399, new size 18446744071973566999, atime 1788201325.000000000, mtime 1788201325.000000000, ctime 1788201325.000000000 00010000:00010000:0.0:1788201325.610864:0:16397:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d48de expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201325.610923:0:16397:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x407:0x0]: losing layout lock 00010000:00010000:0.0:1788201325.610967:0:16397:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d48de expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201325.610974:0:16397:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201325.610977:0:16397:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50cc lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d48de expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788201325.610998:0:16397:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201325.611005:0:16397:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b3b80 x1875064442699520/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201325.611027:0:16397:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b3b80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16397:1875064442699520:192.168.201.130@tcp:36:touch.0 00000100:00100000:0.0:1788201325.611164:0:16397:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448480 going to sleep for 11 seconds 00000100:00100000:0.0:1788201325.627915:0:16397:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b3b80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16397:1875064442699520:192.168.201.130@tcp:36:touch.0 00000080:00200000:0.0:1788201325.627996:0:16397:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x407:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1788201325.628015:0:16397:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x407:0x0](ffff9023449b4b08) 00000002:00010000:0.0:1788201325.628026:0:16397:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x407:0x0]) in obj [0x200000bd1:0x407:0x0], intent: layout flags 0 00010000:00010000:0.0:1788201325.628071:0:16397:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50d3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 16397 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.628081:0:16397:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50d3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16397 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.628091:0:16397:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50d3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16397 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201325.628110:0:16397:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b2680 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16397:1875064442699648:192.168.201.130@tcp:101:touch.0 00000100:00100000:0.0:1788201325.628185:0:16397:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448480 going to sleep for 11 seconds 00000100:00100000:0.0:1788201325.631090:0:16397:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b2680 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16397:1875064442699648:192.168.201.130@tcp:101:touch.0 00010000:00010000:0.0:1788201325.631121:0:16397:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50d3 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d48f3 expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.631135:0:16397:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50d3 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48f3 expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201325.631149:0:16397:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9023488b2680 x1875064442699648/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201341 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201325.631160:0:16397:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50d3 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48f3 expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201325.631173:0:16397:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50d3 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48f3 expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.631190:0:16397:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1031 (0 0) 00000080:00200000:0.0:1788201325.631197:0:16397:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x407:0x0] suppgids 0 0: rc 0 00000080:00010000:0.0:1788201325.631214:0:16397:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x407:0x0](ffff9023449b4b08) for lock 0x4605c7c97e1c50d3 00000080:00010000:0.0:1788201325.631222:0:16397:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c50d3 from it ffffa0b181cdba78 00010000:00010000:0.0:1788201325.631226:0:16397:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50d3 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d48f3 expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201325.631249:0:16397:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50d3 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48f3 expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201325.632413:0:16397:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50d3 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48f3 expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201325.632432:0:16397:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50d3 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48f3 expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201325.632442:0:16397:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x407:0x0](00000000d67d3de0) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50d3 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d48f3 expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201325.632518:0:16397:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x407:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:0.0:1788201325.632537:0:16397:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50d3 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d48f3 expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201325.632547:0:16397:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50d3 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d48f3 expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:0.0:1788201325.632568:0:16397:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:0.0:1788201325.632657:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201325.632673:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442699776:192.168.201.130@tcp:2:touch.0 00000100:00100000:0.0:1788201325.635804:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442699776:192.168.201.130@tcp:2:touch.0 00000080:00200000:2.0:1788201325.635943:0:16397:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x407:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201325.635966:0:16397:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-5:[0x200000bd1:0x407:0x0](ffff9023449b4b08) current size 0, valid attrs 100201f0, mode 9023, uid -2117222832, gid -24399, new size 18446744071973566999, atime 1788201325.000000000, mtime 1788201325.000000000, ctime 1788201325.000000000, rc 0 00000080:00200008:2.0:1788201325.635997:0:16397:0:(file.c:479:ll_file_release()) START file file-5:[0x200000bd1:0x407:0x0](ffff9023449b4b08), flags 104001 00000002:00100000:2.0:1788201325.636052:0:16397:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b2d80 x1875064442699264/t12884901925(12884901925) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201325.636063:0:16397:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201325.636068:0:16397:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c5500 x1875064442700032/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201325.636088:0:16397:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16397:1875064442700032:192.168.201.130@tcp:35:touch.0 00000100:00100000:2.0:1788201325.639497:0:16397:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358f5a480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201325.653228:0:16397:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5500 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16397:1875064442700032:192.168.201.130@tcp:35:touch.0 00000002:00100000:2.0:1788201325.653272:0:16397:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b2d80 x1875064442699264/t12884901925(12884901925) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201325.676843:0:16398:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201325.676872:0:16398:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201325.677669:0:16398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788201325.677687:0:16398:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788201325.677696:0:16398:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201325.678002:0:16398:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201325.678007:0:16398:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:2.0:1788201325.678016:0:16398:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181523bf8 00010000:00010000:2.0:1788201325.678021:0:16398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788201325.678031:0:16398:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788201325.678052:0:16398:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201325.678062:0:16398:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:2.0:1788201325.678069:0:16398:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201325.678086:0:16398:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201325.678091:0:16398:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file-6, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), file ffff902360e25e00, open_flags 8941, mode 81b6 opened 0 00000080:00200000:2.0:1788201325.678099:0:16398:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-6, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=open|creat 00000002:00010000:2.0:1788201325.678123:0:16398:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-6,[0x200000bd1:0x408:0x0]) in obj [0x200000bd1:0x402:0x0], intent: open|creat flags 0104502 00010000:00010000:2.0:1788201325.678184:0:16398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 16398 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.678194:0:16398:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16398 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201325.678209:0:16398:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201325.678224:0:16398:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c5c00 x1875064442700288/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201325.678234:0:16398:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16398 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201325.678257:0:16398:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5c00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16398:1875064442700288:192.168.201.130@tcp:101:touch.0 00000100:00100000:2.0:1788201325.679259:0:16398:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358f5a480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201325.700739:0:16398:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5c00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16398:1875064442700288:192.168.201.130@tcp:101:touch.0 00010000:00010000:2.0:1788201325.700770:0:16398:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d490f expref: -99 pid: 16398 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.700805:0:16398:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d490f expref: -99 pid: 16398 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.700814:0:16398:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d490f expref: -99 pid: 16398 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.700826:0:16398:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d490f expref: -99 pid: 16398 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.700838:0:16398:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d490f expref: -99 pid: 16398 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.700858:0:16398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d490f expref: -99 pid: 16398 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.700867:0:16398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d490f expref: -99 pid: 16398 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.700895:0:16398:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d490f expref: -99 pid: 16398 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201325.700908:0:16398:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023458c5c00 x1875064442700288/t12884901928(12884901928) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201325.700922:0:16398:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458c5c00 x1875064442700288/t12884901928(12884901928) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201325.700936:0:16398:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d490f expref: -99 pid: 16398 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201325.700948:0:16398:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d490f expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.700957:0:16398:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1032 (0 0) 00000080:00200000:2.0:1788201325.700964:0:16398:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201325.700978:0:16398:0:(namei.c:1628:ll_create_it()) VFS Op:name=file-6, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=open|creat 00000080:00200000:2.0:1788201325.701009:0:16398:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x408:0x0](ffff9023449ab248) 00000080:00200000:2.0:1788201325.701128:0:16398:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x408:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788201325.701139:0:16398:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x408:0x0](ffff9023449ab248) 00000080:00010000:2.0:1788201325.701142:0:16398:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x408:0x0](ffff9023449ab248) for lock 0x4605c7c97e1c50da 00000080:00010000:2.0:1788201325.701152:0:16398:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x408:0x0](ffff9023449ab248) 00000080:00200008:2.0:1788201325.701167:0:16398:0:(file.c:1090:ll_file_open()) START file file-6:[0x200000bd1:0x408:0x0](ffff9023449ab248), flags 104501 00000002:00100000:2.0:1788201325.701185:0:16398:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458c5c00 x1875064442700288/t12884901928(12884901928) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788201325.701226:0:16398:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c50da from it ffff902342d49e40 00010000:00010000:2.0:1788201325.701231:0:16398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d490f expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.701239:0:16398:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d490f expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201325.701277:0:16398:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x408:0x0](ffff9023449ab248), inode mode 81a4 mask 40 00000080:00200008:2.0:1788201325.701315:0:16398:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-6:[0x200000bd1:0x408:0x0](ffff9023449ab248) current size 0, valid attrs 20070, mode 9023, uid -2125316528, gid -24399, new size 18446744071973566999, atime 1788201325.000000000, mtime 1788201325.000000000, ctime 1788201325.000000000 00010000:00010000:2.0:1788201325.701335:0:16398:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d490f expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201325.701359:0:16398:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x408:0x0]: losing layout lock 00010000:00010000:2.0:1788201325.701412:0:16398:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d490f expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201325.701419:0:16398:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201325.701421:0:16398:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684000/0x4605c7c97e1c50da lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d490f expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788201325.701447:0:16398:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201325.701454:0:16398:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c6680 x1875064442700544/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201325.701479:0:16398:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16398:1875064442700544:192.168.201.130@tcp:36:touch.0 00000100:00100000:2.0:1788201325.701581:0:16398:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005a480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201325.707008:0:16398:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6680 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16398:1875064442700544:192.168.201.130@tcp:36:touch.0 00000080:00200000:2.0:1788201325.707087:0:16398:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x408:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:2.0:1788201325.707109:0:16398:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x408:0x0](ffff9023449ab248) 00000002:00010000:2.0:1788201325.707120:0:16398:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x408:0x0]) in obj [0x200000bd1:0x408:0x0], intent: layout flags 0 00010000:00010000:2.0:1788201325.707174:0:16398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c50e1 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 16398 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.707184:0:16398:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c50e1 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16398 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.707195:0:16398:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c50e1 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16398 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201325.707215:0:16398:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16398:1875064442700672:192.168.201.130@tcp:101:touch.0 00000100:00100000:2.0:1788201325.707278:0:16398:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005a480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201325.715141:0:16398:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16398:1875064442700672:192.168.201.130@tcp:101:touch.0 00010000:00010000:2.0:1788201325.715175:0:16398:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c50e1 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4924 expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.715192:0:16398:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c50e1 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4924 expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201325.715206:0:16398:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9023458c4700 x1875064442700672/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201341 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201325.715223:0:16398:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c50e1 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4924 expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201325.715342:0:16398:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c50e1 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4924 expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.715364:0:16398:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1032 (0 0) 00000080:00200000:3.0:1788201325.715371:0:16398:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x408:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1788201325.715401:0:16398:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x408:0x0](ffff9023449ab248) for lock 0x4605c7c97e1c50e1 00000080:00010000:3.0:1788201325.715410:0:16398:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c50e1 from it ffffa0b181523a78 00010000:00010000:3.0:1788201325.715415:0:16398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c50e1 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4924 expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.715425:0:16398:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c50e1 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4924 expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201325.715452:0:16398:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c50e1 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4924 expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201325.715464:0:16398:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c50e1 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4924 expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201325.715471:0:16398:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x408:0x0](00000000acc1f1d8) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c50e1 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4924 expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201325.715545:0:16398:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x408:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:3.0:1788201325.715565:0:16398:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c50e1 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4924 expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201325.715572:0:16398:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c50e1 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4924 expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:3.0:1788201325.715592:0:16398:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201325.715677:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201325.715694:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442700800:192.168.201.130@tcp:2:touch.0 00000100:00100000:2.0:1788201325.718217:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442700800:192.168.201.130@tcp:2:touch.0 00000080:00200000:1.0:1788201325.718439:0:16398:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x408:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201325.718468:0:16398:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-6:[0x200000bd1:0x408:0x0](ffff9023449ab248) current size 0, valid attrs 100201f0, mode 9023, uid -2125316528, gid -24399, new size 18446744071973566999, atime 1788201325.000000000, mtime 1788201325.000000000, ctime 1788201325.000000000, rc 0 00000080:00200008:1.0:1788201325.718509:0:16398:0:(file.c:479:ll_file_release()) START file file-6:[0x200000bd1:0x408:0x0](ffff9023449ab248), flags 104001 00000002:00100000:1.0:1788201325.718568:0:16398:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023458c5c00 x1875064442700288/t12884901928(12884901928) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201325.718579:0:16398:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201325.718584:0:16398:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458b9c00 x1875064442701056/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201325.718605:0:16398:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9c00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16398:1875064442701056:192.168.201.130@tcp:35:touch.0 00000100:00100000:1.0:1788201325.718681:0:16398:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fab40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201325.721054:0:16398:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9c00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16398:1875064442701056:192.168.201.130@tcp:35:touch.0 00000002:00100000:1.0:1788201325.721089:0:16398:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023458c5c00 x1875064442700288/t12884901928(12884901928) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201325.730868:0:16399:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201325.730919:0:16399:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201325.730938:0:16399:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788201325.730956:0:16399:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:3.0:1788201325.730964:0:16399:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201325.730973:0:16399:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201325.730976:0:16399:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:3.0:1788201325.730983:0:16399:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181cdbbf8 00010000:00010000:3.0:1788201325.730987:0:16399:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788201325.730997:0:16399:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1788201325.731012:0:16399:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201325.731018:0:16399:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:3.0:1788201325.731023:0:16399:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201325.731034:0:16399:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201325.731038:0:16399:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=file-7, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), file ffff9023460b8d80, open_flags 8941, mode 81b6 opened 0 00000080:00200000:3.0:1788201325.731046:0:16399:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-7, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=open|creat 00000002:00010000:3.0:1788201325.731066:0:16399:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-7,[0x200000bd1:0x409:0x0]) in obj [0x200000bd1:0x402:0x0], intent: open|creat flags 0104502 00010000:00010000:3.0:1788201325.731116:0:16399:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 16399 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201325.731125:0:16399:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16399 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201325.731137:0:16399:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201325.731149:0:16399:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90235268a680 x1875064442701312/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201325.731160:0:16399:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16399 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201325.731183:0:16399:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16399:1875064442701312:192.168.201.130@tcp:101:touch.0 00000100:00100000:3.0:1788201325.731305:0:16399:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe06c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201325.744865:0:16399:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16399:1875064442701312:192.168.201.130@tcp:101:touch.0 00010000:00010000:2.0:1788201325.744920:0:16399:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4940 expref: -99 pid: 16399 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.744943:0:16399:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4940 expref: -99 pid: 16399 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.744952:0:16399:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4940 expref: -99 pid: 16399 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.744963:0:16399:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4940 expref: -99 pid: 16399 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.744973:0:16399:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4940 expref: -99 pid: 16399 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.744987:0:16399:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4940 expref: -99 pid: 16399 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.744996:0:16399:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4940 expref: -99 pid: 16399 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.745010:0:16399:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4940 expref: -99 pid: 16399 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201325.745021:0:16399:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff90235268a680 x1875064442701312/t12884901931(12884901931) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201325.745034:0:16399:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90235268a680 x1875064442701312/t12884901931(12884901931) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201325.745047:0:16399:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4940 expref: -99 pid: 16399 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201325.745059:0:16399:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4940 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.745070:0:16399:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1033 (0 0) 00000080:00200000:2.0:1788201325.745077:0:16399:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201325.745089:0:16399:0:(namei.c:1628:ll_create_it()) VFS Op:name=file-7, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=open|creat 00000080:00200000:2.0:1788201325.745119:0:16399:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x409:0x0](ffff9023449adb88) 00000080:00200000:2.0:1788201325.745236:0:16399:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x409:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788201325.745246:0:16399:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x409:0x0](ffff9023449adb88) 00000080:00010000:2.0:1788201325.745249:0:16399:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x409:0x0](ffff9023449adb88) for lock 0x4605c7c97e1c50e8 00000080:00010000:2.0:1788201325.745258:0:16399:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x409:0x0](ffff9023449adb88) 00000080:00200008:2.0:1788201325.745284:0:16399:0:(file.c:1090:ll_file_open()) START file file-7:[0x200000bd1:0x409:0x0](ffff9023449adb88), flags 104501 00000002:00100000:2.0:1788201325.745303:0:16399:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90235268a680 x1875064442701312/t12884901931(12884901931) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788201325.745356:0:16399:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c50e8 from it ffff9023533eacc0 00010000:00010000:2.0:1788201325.745362:0:16399:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4940 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.745371:0:16399:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4940 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201325.745401:0:16399:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x409:0x0](ffff9023449adb88), inode mode 81a4 mask 40 00000080:00200008:2.0:1788201325.745437:0:16399:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-7:[0x200000bd1:0x409:0x0](ffff9023449adb88) current size 0, valid attrs 20070, mode 9023, uid -2117222832, gid -24399, new size 18446744071973566999, atime 1788201325.000000000, mtime 1788201325.000000000, ctime 1788201325.000000000 00010000:00010000:2.0:1788201325.745455:0:16399:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d4940 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201325.745478:0:16399:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x409:0x0]: losing layout lock 00010000:00010000:2.0:1788201325.746962:0:16399:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4940 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201325.746976:0:16399:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201325.746979:0:16399:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c50e8 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4940 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788201325.747005:0:16399:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201325.747016:0:16399:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c7480 x1875064442701568/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201325.747041:0:16399:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c7480 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16399:1875064442701568:192.168.201.130@tcp:36:touch.0 00000100:00100000:2.0:1788201325.747122:0:16399:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023587406c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201325.754320:0:16399:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c7480 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16399:1875064442701568:192.168.201.130@tcp:36:touch.0 00000080:00200000:2.0:1788201325.754419:0:16399:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x409:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:2.0:1788201325.754443:0:16399:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x409:0x0](ffff9023449adb88) 00000002:00010000:2.0:1788201325.754454:0:16399:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x409:0x0]) in obj [0x200000bd1:0x409:0x0], intent: layout flags 0 00010000:00010000:2.0:1788201325.754519:0:16399:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c50ef lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 16399 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.754528:0:16399:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c50ef lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16399 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.754538:0:16399:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c50ef lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16399 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201325.754558:0:16399:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4a80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16399:1875064442701696:192.168.201.130@tcp:101:touch.0 00000100:00100000:2.0:1788201325.754628:0:16399:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023587406c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201325.758542:0:16399:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4a80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16399:1875064442701696:192.168.201.130@tcp:101:touch.0 00010000:00010000:2.0:1788201325.758572:0:16399:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c50ef lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4955 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.758586:0:16399:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c50ef lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4955 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201325.758601:0:16399:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9023458c4a80 x1875064442701696/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201341 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201325.758612:0:16399:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c50ef lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4955 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201325.758623:0:16399:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c50ef lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4955 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.758641:0:16399:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1033 (0 0) 00000080:00200000:2.0:1788201325.758647:0:16399:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x409:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788201325.758664:0:16399:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x409:0x0](ffff9023449adb88) for lock 0x4605c7c97e1c50ef 00000080:00010000:2.0:1788201325.758672:0:16399:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c50ef from it ffffa0b181cdba78 00010000:00010000:2.0:1788201325.758675:0:16399:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c50ef lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4955 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201325.758685:0:16399:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c50ef lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4955 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201325.758708:0:16399:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c50ef lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4955 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201325.758733:0:16399:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c50ef lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4955 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788201325.758741:0:16399:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x409:0x0](0000000002670e8b) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c50ef lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4955 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201325.758809:0:16399:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x409:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:2.0:1788201325.758830:0:16399:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c50ef lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4955 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201325.758838:0:16399:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c50ef lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4955 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1788201325.758858:0:16399:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1788201325.758976:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201325.758990:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442701824:192.168.201.130@tcp:2:touch.0 00000100:00100000:2.0:1788201325.763088:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442701824:192.168.201.130@tcp:2:touch.0 00000080:00200000:3.0:1788201325.763224:0:16399:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x409:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201325.763248:0:16399:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-7:[0x200000bd1:0x409:0x0](ffff9023449adb88) current size 0, valid attrs 100201f0, mode 9023, uid -2117222832, gid -24399, new size 18446744071973566999, atime 1788201325.000000000, mtime 1788201325.000000000, ctime 1788201325.000000000, rc 0 00000080:00200008:3.0:1788201325.763287:0:16399:0:(file.c:479:ll_file_release()) START file file-7:[0x200000bd1:0x409:0x0](ffff9023449adb88), flags 104001 00000002:00100000:3.0:1788201325.763353:0:16399:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff90235268a680 x1875064442701312/t12884901931(12884901931) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201325.763366:0:16399:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201325.763371:0:16399:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352688380 x1875064442702080/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201325.763402:0:16399:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16399:1875064442702080:192.168.201.130@tcp:35:touch.0 00000100:00100000:3.0:1788201325.763477:0:16399:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe03c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201325.766644:0:16399:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16399:1875064442702080:192.168.201.130@tcp:35:touch.0 00000002:00100000:2.0:1788201325.766688:0:16399:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff90235268a680 x1875064442701312/t12884901931(12884901931) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201341 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788201326.209450:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788201326.442003:0:16412:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201326.442045:0:16412:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201326.442073:0:16412:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788201326.442093:0:16412:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1788201326.442103:0:16412:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201326.442112:0:16412:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201326.442116:0:16412:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:3.0:1788201326.442127:0:16412:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b180d4bb58 00010000:00010000:3.0:1788201326.442132:0:16412:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788201326.442143:0:16412:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1788201326.442164:0:16412:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201326.442173:0:16412:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:3.0:1788201326.442179:0:16412:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201326.442190:0:16412:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-2, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), flags=9 00000080:00200000:3.0:1788201326.442195:0:16412:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-2, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=getattr 00000002:00010000:3.0:1788201326.442209:0:16412:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-2,[0x0:0x0:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201326.442268:0:16412:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 16412 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.442277:0:16412:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16412 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.442289:0:16412:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16412 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201326.442328:0:16412:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689f80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16412:1875064442702208:192.168.201.130@tcp:101:chmod.0 00000080:00200000:2.0:1788201326.446012:0:16414:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201326.446052:0:16414:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201326.446079:0:16414:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788201326.446100:0:16414:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788201326.446109:0:16414:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201326.446119:0:16414:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201326.446123:0:16414:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:2.0:1788201326.446132:0:16414:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181c1bb58 00010000:00010000:2.0:1788201326.446136:0:16414:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788201326.446147:0:16414:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1788201326.446166:0:16414:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201326.446175:0:16414:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:2.0:1788201326.446181:0:16414:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201326.446193:0:16414:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-4, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), flags=9 00000080:00200000:2.0:1788201326.446197:0:16414:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-4, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=getattr 00000002:00010000:2.0:1788201326.446210:0:16414:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-4,[0x0:0x0:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201326.446339:0:16414:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 16414 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.446348:0:16414:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 16414 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.446431:0:16414:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 16414 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201326.446463:0:16414:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16414:1875064442702336:192.168.201.130@tcp:101:chmod.0 00000100:00100000:2.0:1788201326.446568:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023587406c0 going to sleep for 11 seconds 00000080:00200000:1.0:1788201326.448085:0:16415:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201326.448111:0:16415:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201326.448130:0:16415:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788201326.448146:0:16415:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788201326.448155:0:16415:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201326.448163:0:16415:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201326.448167:0:16415:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:1.0:1788201326.448175:0:16415:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181c23b58 00010000:00010000:1.0:1788201326.448179:0:16415:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788201326.448187:0:16415:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788201326.448203:0:16415:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201326.448210:0:16415:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:1.0:1788201326.448214:0:16415:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201326.448223:0:16415:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-5, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), flags=9 00000080:00200000:1.0:1788201326.448227:0:16415:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-5, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=getattr 00000002:00010000:1.0:1788201326.448237:0:16415:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-5,[0x0:0x0:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201326.448283:0:16415:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 16415 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.448296:0:16415:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 16415 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.448306:0:16415:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 16415 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201326.448333:0:16415:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9f80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16415:1875064442702464:192.168.201.130@tcp:101:chmod.0 00000100:00100000:3.0:1788201326.448758:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0480 going to sleep for 11 seconds 00000080:00200000:2.0:1788201326.449820:0:16413:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201326.449842:0:16413:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201326.449857:0:16413:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:2.0:1788201326.449874:0:16413:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:2.0:1788201326.449907:0:16413:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201326.449915:0:16413:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201326.449919:0:16413:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:2.0:1788201326.449926:0:16413:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181c13b58 00010000:00010000:2.0:1788201326.449930:0:16413:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:2.0:1788201326.449944:0:16413:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:2.0:1788201326.449958:0:16413:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201326.449965:0:16413:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:2.0:1788201326.449971:0:16413:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201326.449982:0:16413:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-3, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), flags=9 00000080:00200000:2.0:1788201326.449985:0:16413:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-3, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=getattr 00000002:00010000:2.0:1788201326.449996:0:16413:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-3,[0x0:0x0:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201326.450042:0:16413:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 6 type: IBT flags: 0x10000000000000 pid: 16413 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.450051:0:16413:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 6 type: IBT flags: 0x0 pid: 16413 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.450060:0:16413:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 6 type: IBT flags: 0x0 pid: 16413 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201326.450085:0:16413:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6d80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16413:1875064442702592:192.168.201.130@tcp:101:chmod.0 00000100:00100000:2.0:1788201326.450180:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234366d300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201326.456104:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fab40 going to sleep for 11 seconds 00000080:00200000:3.0:1788201326.460071:0:16417:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201326.460113:0:16417:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201326.460139:0:16417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1788201326.460160:0:16417:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:3.0:1788201326.460171:0:16417:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201326.460180:0:16417:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201326.460184:0:16417:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:3.0:1788201326.460194:0:16417:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181c33b58 00010000:00010000:3.0:1788201326.460199:0:16417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1788201326.460210:0:16417:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:3.0:1788201326.460231:0:16417:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201326.460240:0:16417:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:3.0:1788201326.460247:0:16417:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201326.460258:0:16417:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-7, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), flags=9 00000080:00200000:3.0:1788201326.460263:0:16417:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-7, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=getattr 00000002:00010000:3.0:1788201326.460278:0:16417:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-7,[0x0:0x0:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201326.460347:0:16417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 7 type: IBT flags: 0x10000000000000 pid: 16417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.460356:0:16417:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 7 type: IBT flags: 0x0 pid: 16417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.460379:0:16417:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 7 type: IBT flags: 0x0 pid: 16417 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201326.460418:0:16417:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16417:1875064442702720:192.168.201.130@tcp:101:chmod.0 00000100:00100000:3.0:1788201326.460534:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0180 going to sleep for 11 seconds 00000080:00200000:1.0:1788201326.461660:0:16411:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201326.461681:0:16411:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201326.461697:0:16411:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1788201326.461712:0:16411:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:1.0:1788201326.461721:0:16411:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201326.461729:0:16411:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201326.461732:0:16411:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:1.0:1788201326.461739:0:16411:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181c03b58 00010000:00010000:1.0:1788201326.461743:0:16411:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1788201326.461753:0:16411:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:1.0:1788201326.461768:0:16411:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201326.461776:0:16411:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:1.0:1788201326.461781:0:16411:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201326.461792:0:16411:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-1, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), flags=9 00000080:00200000:1.0:1788201326.461796:0:16411:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-1, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=getattr 00000002:00010000:1.0:1788201326.461805:0:16411:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-1,[0x0:0x0:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201326.461849:0:16411:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 8 type: IBT flags: 0x10000000000000 pid: 16411 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.461858:0:16411:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 8 type: IBT flags: 0x0 pid: 16411 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.461869:0:16411:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 8 type: IBT flags: 0x0 pid: 16411 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788201326.461888:0:16416:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201326.461900:0:16416:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201326.461909:0:16416:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000100:00100000:1.0:1788201326.461910:0:16411:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458bad80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16411:1875064442702848:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201326.461917:0:16416:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:2.0:1788201326.461928:0:16416:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201326.461932:0:16416:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201326.461935:0:16416:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:2.0:1788201326.461938:0:16416:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181c2bb58 00010000:00010000:2.0:1788201326.461941:0:16416:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1788201326.461948:0:16416:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:2.0:1788201326.461957:0:16416:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201326.461961:0:16416:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:2.0:1788201326.461964:0:16416:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201326.461972:0:16416:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-6, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), flags=9 00000080:00200000:2.0:1788201326.461975:0:16416:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-6, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=getattr 00000002:00010000:2.0:1788201326.461981:0:16416:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-6,[0x0:0x0:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 01 00000100:00100000:1.0:1788201326.461988:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa180 going to sleep for 11 seconds 00010000:00010000:2.0:1788201326.462002:0:16416:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 9 type: IBT flags: 0x10000000000000 pid: 16416 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.462009:0:16416:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 9 type: IBT flags: 0x0 pid: 16416 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.462016:0:16416:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 9 type: IBT flags: 0x0 pid: 16416 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201326.462027:0:16416:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16416:1875064442702976:192.168.201.130@tcp:101:chmod.0 00000100:00100000:2.0:1788201326.462054:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201326.466839:0:16412:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689f80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16412:1875064442702208:192.168.201.130@tcp:101:chmod.0 00010000:00010000:0.0:1788201326.466919:0:16412:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 9 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.466941:0:16412:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.466950:0:16412:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.466963:0:16412:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.466975:0:16412:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.466991:0:16412:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.467000:0:16412:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.467009:0:16412:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201326.467030:0:16412:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352689f80 x1875064442702208/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201342 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201326.467043:0:16412:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201326.467057:0:16412:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.467068:0:16412:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1028 (0 0) 00000080:00200000:0.0:1788201326.467077:0:16412:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201326.467186:0:16412:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x404:0x0](ffff9023449a7448): rc = 0 00000080:00200000:0.0:1788201326.467210:0:16412:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x404:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201326.467222:0:16412:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x404:0x0](ffff9023449a7448) for lock 0x4605c7c97e1c50f6 00000080:00010000:0.0:1788201326.467234:0:16412:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x404:0x0](ffff9023449a7448) 00000080:00010000:0.0:1788201326.467238:0:16412:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c50f6 from it ffffa0b180d4bb20 00010000:00010000:0.0:1788201326.467242:0:16412:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.467259:0:16412:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201326.467303:0:16412:0:(file.c:6283:ll_getattr_dentry()) START file file-2:[0x200000bd1:0x404:0x0](ffff9023449a7448), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201326.467312:0:16412:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x404:0x0](ffff9023449a7448),name=file-2 00000002:00010000:0.0:1788201326.467336:0:16412:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x404:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201326.467345:0:16412:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.467358:0:16412:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201326.467376:0:16412:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201326.467381:0:16412:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x404:0x0](ffff9023449a7448) 00000080:00010000:0.0:1788201326.467385:0:16412:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x404:0x0](ffff9023449a7448) for lock 0x4605c7c97e1c50f6 00000080:00010000:0.0:1788201326.467389:0:16412:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c50f6 from it ffffa0b180d4bcb8 00010000:00010000:0.0:1788201326.467392:0:16412:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.467400:0:16412:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201326.467429:0:16412:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x404:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201326.467441:0:16412:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x404:0x0] 00000008:00010000:0.0:1788201326.467475:0:16412:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201326.467480:0:16412:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff90235306dbb0, osc lock ffff902343fb7a20, flags 0x400001002 00010000:00010000:0.0:1788201326.467536:0:16412:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.467548:0:16412:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788201326.467587:0:16412:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1788201326.467611:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442703104:192.168.201.130@tcp:101:chmod.0 00000100:00100000:2.0:1788201326.472509:0:16414:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16414:1875064442702336:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201326.472549:0:16414:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 8 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201326.472553:0:16413:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6d80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16413:1875064442702592:192.168.201.130@tcp:101:chmod.0 00010000:00010000:0.0:1788201326.472566:0:16413:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 8 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.472572:0:16414:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.472575:0:16413:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.472581:0:16414:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.472582:0:16413:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.472594:0:16413:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.472594:0:16414:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.472607:0:16413:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.472607:0:16414:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.472623:0:16413:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.472623:0:16414:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.472634:0:16413:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.472634:0:16414:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.472643:0:16413:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.472643:0:16414:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201326.472656:0:16413:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c6d80 x1875064442702592/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201342 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201326.472656:0:16414:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c6a00 x1875064442702336/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201342 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201326.472669:0:16413:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201326.472669:0:16414:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201326.472685:0:16413:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201326.472685:0:16414:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.472696:0:16413:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1029 (0 0) 00010000:00010000:2.0:1788201326.472696:0:16414:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1030 (0 0) 00000080:00200000:0.0:1788201326.472705:0:16413:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201326.472705:0:16414:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201326.472817:0:16413:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x405:0x0](ffff9023449b5348): rc = 0 00000080:00200000:2.0:1788201326.472817:0:16414:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x406:0x0](ffff902344979148): rc = 0 00000080:00200000:2.0:1788201326.472839:0:16414:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x406:0x0]: layout version change: 0 -> 0 00000080:00200000:0.0:1788201326.472840:0:16413:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x405:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201326.472851:0:16413:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x405:0x0](ffff9023449b5348) for lock 0x4605c7c97e1c510b 00000080:00010000:2.0:1788201326.472851:0:16414:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x406:0x0](ffff902344979148) for lock 0x4605c7c97e1c50fd 00000080:00010000:2.0:1788201326.472864:0:16414:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x406:0x0](ffff902344979148) 00000080:00010000:0.0:1788201326.472868:0:16413:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x405:0x0](ffff9023449b5348) 00000080:00010000:2.0:1788201326.472868:0:16414:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c50fd from it ffffa0b181c1bb20 00000080:00010000:0.0:1788201326.472870:0:16413:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c510b from it ffffa0b181c13b20 00010000:00010000:2.0:1788201326.472872:0:16414:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.472874:0:16413:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.472950:0:16414:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788201326.473000:0:16414:0:(file.c:6283:ll_getattr_dentry()) START file file-4:[0x200000bd1:0x406:0x0](ffff902344979148), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201326.473009:0:16414:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x406:0x0](ffff902344979148),name=file-4 00000002:00010000:2.0:1788201326.473032:0:16414:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x406:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201326.473041:0:16414:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201326.473052:0:16414:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201326.473059:0:16414:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201326.473064:0:16414:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x406:0x0](ffff902344979148) 00000080:00010000:2.0:1788201326.473067:0:16414:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x406:0x0](ffff902344979148) for lock 0x4605c7c97e1c50fd 00000080:00010000:2.0:1788201326.473071:0:16414:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c50fd from it ffffa0b181c1bcb8 00010000:00010000:2.0:1788201326.473075:0:16414:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201326.473082:0:16414:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201326.473111:0:16414:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x406:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201326.473123:0:16414:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x406:0x0] 00010000:00010000:0.0:1788201326.473129:0:16413:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201326.473150:0:16413:0:(file.c:6283:ll_getattr_dentry()) START file file-3:[0x200000bd1:0x405:0x0](ffff9023449b5348), request_mask 2047, flags 0, foreign 0 00000008:00010000:2.0:1788201326.473158:0:16414:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000080:00200000:0.0:1788201326.473161:0:16413:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x405:0x0](ffff9023449b5348),name=file-3 00000008:00010000:2.0:1788201326.473163:0:16414:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398fb0, osc lock ffff9023500dd7c0, flags 0x400001002 00000002:00010000:0.0:1788201326.473169:0:16413:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x405:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201326.473175:0:16413:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.473183:0:16413:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201326.473188:0:16413:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201326.473193:0:16413:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x405:0x0](ffff9023449b5348) 00000080:00010000:0.0:1788201326.473195:0:16413:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x405:0x0](ffff9023449b5348) for lock 0x4605c7c97e1c510b 00000080:00010000:0.0:1788201326.473198:0:16413:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c510b from it ffffa0b181c13cb8 00010000:00010000:0.0:1788201326.473200:0:16413:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.473207:0:16413:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201326.473215:0:16414:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 3/1,0 mode: --/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1788201326.473225:0:16413:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x405:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00010000:00010000:2.0:1788201326.473227:0:16414:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 3/1,0 mode: --/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000001:00010000:0.0:1788201326.473229:0:16413:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x405:0x0] 00000020:00200000:2.0:1788201326.473273:0:16414:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1788201326.473299:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442703232:192.168.201.130@tcp:101:chmod.0 00000008:00010000:0.0:1788201326.473422:0:16413:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201326.473425:0:16413:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff90235306deb0, osc lock ffff902343fb6130, flags 0x400001002 00010000:00010000:0.0:1788201326.473454:0:16413:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.473462:0:16413:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201326.473518:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442703360:192.168.201.130@tcp:101:chmod.0 00000020:00200000:0.0:1788201326.473729:0:16413:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1788201326.478219:0:16415:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9f80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16415:1875064442702464:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201326.478266:0:16415:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.478289:0:16415:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.478303:0:16415:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.478315:0:16415:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.478327:0:16415:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.478341:0:16415:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.478361:0:16415:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.478383:0:16415:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201326.478400:0:16415:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458b9f80 x1875064442702464/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201342 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201326.478414:0:16415:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201326.478430:0:16415:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.478448:0:16415:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1031 (0 0) 00000080:00200000:2.0:1788201326.478457:0:16415:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201326.478589:0:16415:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x407:0x0](ffff9023449b4b08): rc = 0 00000080:00200000:2.0:1788201326.478615:0:16415:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x407:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201326.478628:0:16415:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x407:0x0](ffff9023449b4b08) for lock 0x4605c7c97e1c5104 00000080:00010000:2.0:1788201326.478642:0:16415:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x407:0x0](ffff9023449b4b08) 00000080:00010000:2.0:1788201326.478646:0:16415:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5104 from it ffffa0b181c23b20 00010000:00010000:2.0:1788201326.478650:0:16415:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.478661:0:16415:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788201326.478710:0:16415:0:(file.c:6283:ll_getattr_dentry()) START file file-5:[0x200000bd1:0x407:0x0](ffff9023449b4b08), request_mask 2047, flags 0, foreign 0 00000100:00100000:2.0:1788201326.483186:0:16417:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16417:1875064442702720:192.168.201.130@tcp:101:chmod.0 00000100:00100000:3.0:1788201326.483186:0:16416:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16416:1875064442702976:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201326.483220:0:16417:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.483220:0:16416:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.483240:0:16416:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.483241:0:16417:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.483250:0:16416:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.483263:0:16416:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.483275:0:16416:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.483290:0:16416:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.483305:0:16416:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.483313:0:16416:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201326.483329:0:16416:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458c5880 x1875064442702976/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201342 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788201326.483342:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201326.483343:0:16416:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.483435:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.483451:0:2360:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.483527:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.483538:0:16417:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.483540:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 6/2,0 mode: --/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.483548:0:16417:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.483550:0:2362:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.483558:0:16417:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.483560:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.483567:0:16417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.483574:0:16417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1788201326.483578:0:2362:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.483581:0:16417:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201326.483591:0:16417:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352689180 x1875064442702720/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201342 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201326.483599:0:16417:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788201326.483600:0:2362:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1788201326.483611:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201326.483613:0:16417:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.483620:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.483626:0:16417:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1033 (0 0) 00010000:00010000:3.0:1788201326.483626:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.483635:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201326.483636:0:16417:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000100:00100000:3.0:1788201326.483656:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442703232:192.168.201.130@tcp:101:chmod.0 00000080:00200000:3.0:1788201326.483709:0:16414:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x406:0x0] updating i_size 0 i_blocks 1 00000080:00200000:2.0:1788201326.483763:0:16417:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x409:0x0](ffff9023449adb88): rc = 0 00000080:00200000:2.0:1788201326.483788:0:16417:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x409:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201326.483801:0:16417:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x409:0x0](ffff9023449adb88) for lock 0x4605c7c97e1c5112 00000080:00010000:2.0:1788201326.483817:0:16417:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x409:0x0](ffff9023449adb88) 00000080:00010000:2.0:1788201326.483822:0:16417:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5112 from it ffffa0b181c33b20 00010000:00010000:2.0:1788201326.483826:0:16417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.483835:0:16417:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788201326.483905:0:16417:0:(file.c:6283:ll_getattr_dentry()) START file file-7:[0x200000bd1:0x409:0x0](ffff9023449adb88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201326.483916:0:16417:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x409:0x0](ffff9023449adb88),name=file-7 00000020:00010000:3.0:1788201326.483933:0:16414:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc6bf0 (ffff902350b9d998/0) at cl_lock_release():245 00000020:00010000:3.0:1788201326.483937:0:16414:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc6bf0 (ffff902350b9d998/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201326.483943:0:16414:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398fb0 (ffff902350b9d908/1) at cl_lock_cancel():142 00000002:00010000:2.0:1788201326.483945:0:16417:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x409:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 0 00000020:00010000:3.0:1788201326.483948:0:16414:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc6bf0 (ffff902350b9d998/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201326.483951:0:16414:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398fb0 (ffff902350b9d998/1) at cl_lock_fini():72 00010000:00010000:2.0:1788201326.483956:0:16417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201326.483968:0:16417:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201326.483975:0:16417:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201326.483980:0:16417:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x409:0x0](ffff9023449adb88) 00000080:00010000:2.0:1788201326.483984:0:16417:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x409:0x0](ffff9023449adb88) for lock 0x4605c7c97e1c5112 00000080:00200000:3.0:1788201326.483986:0:16414:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x406:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00010000:2.0:1788201326.483987:0:16417:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5112 from it ffffa0b181c33cb8 00010000:00010000:2.0:1788201326.483990:0:16417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201326.483997:0:16417:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201326.484024:0:16417:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x409:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201326.484039:0:16417:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x409:0x0] 00000008:00010000:2.0:1788201326.484081:0:16417:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201326.484085:0:16417:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398c30, osc lock ffff9023500dc000, flags 0x400001002 00000080:00200000:3.0:1788201326.484089:0:16414:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201326.484100:0:16414:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201326.484112:0:16414:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:3.0:1788201326.484121:0:16414:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:3.0:1788201326.484127:0:16414:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201326.484131:0:16414:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201326.484134:0:16414:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:3.0:1788201326.484137:0:16414:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181c1bc08 00010000:00010000:3.0:1788201326.484141:0:16414:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1788201326.484147:0:16417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 3/1,0 mode: --/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.484148:0:16414:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:2.0:1788201326.484157:0:16417:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 3/1,0 mode: --/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1788201326.484158:0:16414:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201326.484166:0:16414:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:3.0:1788201326.484172:0:16414:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201326.484176:0:16414:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-4, flags=9 00000020:00200000:2.0:1788201326.484189:0:16417:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1788201326.484218:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442703488:192.168.201.130@tcp:101:chmod.0 00000002:00010000:2.0:1788201326.484228:0:16416:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.484238:0:16416:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1032 (0 0) 00000080:00200000:2.0:1788201326.484243:0:16416:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1788201326.484344:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788201326.484455:0:2360:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788201326.484466:0:2360:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1788201326.484482:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.484490:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.484497:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.484506:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788201326.484524:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442703104:192.168.201.130@tcp:101:chmod.0 00000080:00200000:0.0:1788201326.484803:0:16412:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x404:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201326.484808:0:16412:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372e40 (ffff902348a2fab8/0) at cl_lock_release():245 00000020:00010000:0.0:1788201326.484811:0:16412:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372e40 (ffff902348a2fab8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201326.484815:0:16412:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235306dbb0 (ffff902348a2f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201326.484819:0:16412:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372e40 (ffff902348a2fab8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201326.484821:0:16412:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235306dbb0 (ffff902348a2fab8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201326.484857:0:16412:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x404:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201326.484922:0:16412:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201326.484933:0:16412:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201326.484943:0:16412:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201326.484954:0:16412:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:0.0:1788201326.484960:0:16412:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201326.484964:0:16412:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201326.484967:0:16412:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:0.0:1788201326.484972:0:16412:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b180d4bc08 00010000:00010000:0.0:1788201326.484974:0:16412:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201326.484982:0:16412:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:0.0:1788201326.484992:0:16412:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201326.484997:0:16412:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:0.0:1788201326.485000:0:16412:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201326.485003:0:16412:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-2, flags=9 00010000:00010000:0.0:1788201326.485187:0:2359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.485209:0:2359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.485223:0:2359:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.485235:0:2359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788201326.485246:0:2359:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788201326.485265:0:2359:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1788201326.485281:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.485291:0:2359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.485298:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.485307:0:2359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788201326.485326:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442703360:192.168.201.130@tcp:101:chmod.0 00000080:00200000:2.0:1788201326.486513:0:16413:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x405:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201326.486525:0:16413:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9023463704a0 (ffff902348a2fc68/0) at cl_lock_release():245 00000020:00010000:2.0:1788201326.486530:0:16413:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023463704a0 (ffff902348a2fc68/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201326.486536:0:16413:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235306deb0 (ffff902348a2f488/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201326.486544:0:16413:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023463704a0 (ffff902348a2fc68/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201326.486547:0:16413:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235306deb0 (ffff902348a2fc68/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201326.486588:0:16413:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x405:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:1.0:1788201326.486599:0:16411:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458bad80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16411:1875064442702848:192.168.201.130@tcp:101:chmod.0 00010000:00010000:1.0:1788201326.486623:0:16411:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.486641:0:16411:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.486649:0:16411:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788201326.486650:0:16413:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00010000:00010000:1.0:1788201326.486659:0:16411:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201326.486666:0:16413:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201326.486670:0:16411:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.486679:0:16413:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:1.0:1788201326.486681:0:16411:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.486689:0:16411:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.486691:0:16413:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:1.0:1788201326.486697:0:16411:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788201326.486705:0:16413:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000002:00100000:1.0:1788201326.486710:0:16411:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458bad80 x1875064442702848/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201342 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788201326.486713:0:16413:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201326.486716:0:16413:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:2.0:1788201326.486722:0:16413:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181c13c08 00010000:00010000:2.0:1788201326.486726:0:16413:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:2.0:1788201326.486734:0:16413:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:2.0:1788201326.486746:0:16413:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201326.486752:0:16413:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:2.0:1788201326.486757:0:16413:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201326.486761:0:16413:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-3, flags=9 00000002:00010000:1.0:1788201326.486816:0:16411:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201326.486831:0:16411:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.486840:0:16411:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1027 (0 0) 00000080:00200000:1.0:1788201326.486845:0:16411:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201326.486863:0:16416:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x408:0x0](ffff9023449ab248): rc = 0 00000080:00200000:2.0:1788201326.486902:0:16416:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x408:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201326.486915:0:16416:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x408:0x0](ffff9023449ab248) for lock 0x4605c7c97e1c5120 00000080:00010000:2.0:1788201326.486927:0:16416:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x408:0x0](ffff9023449ab248) 00000080:00010000:2.0:1788201326.486929:0:16416:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5120 from it ffffa0b181c2bb20 00010000:00010000:2.0:1788201326.486933:0:16416:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.486942:0:16416:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788201326.486968:0:16416:0:(file.c:6283:ll_getattr_dentry()) START file file-6:[0x200000bd1:0x408:0x0](ffff9023449ab248), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201326.486976:0:16416:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x408:0x0](ffff9023449ab248),name=file-6 00000002:00010000:2.0:1788201326.486989:0:16416:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x408:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201326.486997:0:16416:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201326.487005:0:16416:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201326.487012:0:16416:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201326.487016:0:16416:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x408:0x0](ffff9023449ab248) 00000080:00010000:2.0:1788201326.487019:0:16416:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x408:0x0](ffff9023449ab248) for lock 0x4605c7c97e1c5120 00000080:00010000:2.0:1788201326.487022:0:16416:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5120 from it ffffa0b181c2bcb8 00010000:00010000:2.0:1788201326.487025:0:16416:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201326.487032:0:16416:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201326.487056:0:16416:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x408:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201326.487065:0:16416:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x408:0x0] 00000008:00010000:2.0:1788201326.487093:0:16416:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201326.487097:0:16416:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398ab0, osc lock ffff9023500dce40, flags 0x400001002 00010000:00010000:2.0:1788201326.487148:0:16416:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.487155:0:2361:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.487159:0:16416:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.487168:0:2361:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 6/2,0 mode: --/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.487186:0:2361:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201326.487192:0:16416:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788201326.487197:0:2361:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1788201326.487211:0:2361:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788201326.487220:0:2361:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1788201326.487229:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.487237:0:2361:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.487245:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201326.487257:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442703616:192.168.201.130@tcp:101:chmod.0 00000080:00200000:1.0:1788201326.487355:0:16411:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x403:0x0](ffff90234497ec08): rc = 0 00000080:00200000:1.0:1788201326.487384:0:16411:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x403:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1788201326.487395:0:16411:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x403:0x0](ffff90234497ec08) for lock 0x4605c7c97e1c5119 00000080:00010000:1.0:1788201326.487402:0:16411:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x403:0x0](ffff90234497ec08) 00000080:00010000:1.0:1788201326.487406:0:16411:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5119 from it ffffa0b181c03b20 00010000:00010000:1.0:1788201326.487410:0:16411:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.487419:0:16411:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788201326.487440:0:16411:0:(file.c:6283:ll_getattr_dentry()) START file file-1:[0x200000bd1:0x403:0x0](ffff90234497ec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201326.487445:0:16411:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x403:0x0](ffff90234497ec08),name=file-1 00000002:00010000:1.0:1788201326.487454:0:16411:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x403:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201326.487462:0:16411:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201326.487541:0:16411:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201326.487550:0:16411:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201326.487554:0:16411:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x403:0x0](ffff90234497ec08) 00000080:00010000:1.0:1788201326.487556:0:16411:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x403:0x0](ffff90234497ec08) for lock 0x4605c7c97e1c5119 00000080:00010000:1.0:1788201326.487559:0:16411:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5119 from it ffffa0b181c03cb8 00010000:00010000:1.0:1788201326.487562:0:16411:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201326.487569:0:16411:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201326.487588:0:16411:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x403:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788201326.487595:0:16411:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x403:0x0] 00000008:00010000:1.0:1788201326.487617:0:16411:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788201326.487620:0:16411:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023450454b0, osc lock ffff90235090ce40, flags 0x400001002 00010000:00010000:1.0:1788201326.487647:0:16411:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.487656:0:16411:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788201326.487681:0:16411:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1788201326.487695:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442703744:192.168.201.130@tcp:101:chmod.0 00000080:00200008:0.0:1788201326.487832:0:16412:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-2:[0x200000bd1:0x404:0x0](ffff9023449a7448) current size 0, valid attrs 41, mode 8180, uid 1117585408, gid -28637, new size 0, atime 1788201326.000000000, mtime 1788201326.000000000, ctime 1788201326.000000000 00010000:00010000:0.0:1788201326.487855:0:16412:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201326.487874:0:16412:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c50a9 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4860 expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201326.487911:0:2361:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.487923:0:16412:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1788201326.487924:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442703488:192.168.201.130@tcp:101:chmod.0 00010000:00010000:0.0:1788201326.487930:0:16412:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201326.487933:0:16412:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c50f6 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4971 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201326.487953:0:16412:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201326.487962:0:16412:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b1180 x1875064442704000/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000080:00200008:3.0:1788201326.487964:0:16414:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-4:[0x200000bd1:0x406:0x0](ffff902344979148) current size 0, valid attrs 41, mode 8180, uid 1094287360, gid -28637, new size 0, atime 1788201326.000000000, mtime 1788201326.000000000, ctime 1788201326.000000000 00000100:00100000:0.0:1788201326.487977:0:16412:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1180 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16412:1875064442704000:192.168.201.130@tcp:36:chmod.0 00010000:00010000:3.0:1788201326.487980:0:16414:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201326.487992:0:16414:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c50c5 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d48c2 expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201326.488013:0:16414:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201326.488020:0:16414:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201326.488080:0:16414:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685a00/0x4605c7c97e1c50fd lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d498d expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201326.488093:0:16414:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 2 is allocated opc 36, max 7 00000100:00100000:3.0:1788201326.488098:0:16414:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352689500 x1875064442704256/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201326.488111:0:16414:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16414:1875064442704256:192.168.201.130@tcp:36:chmod.0 00000100:00100000:0.0:1788201326.489465:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 11 seconds 00010000:00010000:0.0:1788201326.489551:0:16421:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201326.489567:0:16421:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1788201326.489578:0:16421:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c510b from it ffffa0b181c4bd78 00010000:00010000:0.0:1788201326.489583:0:16421:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201326.489592:0:16421:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201326.489662:0:16421:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5151 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.489671:0:16421:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5151 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.489696:0:16421:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.489705:0:16421:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201326.489711:0:16421:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5104 from it ffffa0b181c4bd78 00010000:00010000:0.0:1788201326.489715:0:16421:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.489724:0:16421:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201326.489745:0:16421:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201326.489754:0:16421:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1788201326.489761:0:16421:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5120 from it ffffa0b181c4bd78 00010000:00010000:0.0:1788201326.489764:0:16421:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201326.489771:0:16421:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201326.489789:0:16421:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201326.489796:0:16421:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1788201326.489802:0:16421:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5112 from it ffffa0b181c4bd78 00010000:00010000:0.0:1788201326.489805:0:16421:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201326.489812:0:16421:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201326.489837:0:16421:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5158 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.489844:0:16421:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5158 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.489865:0:16421:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c515f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.489872:0:16421:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c515f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.489925:0:16421:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c5166 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 6 type: IBT flags: 0x10000000000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.489932:0:16421:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c5166 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1788201326.490021:0:16421:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x407:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201326.490033:0:16421:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x407:0x0] 00000008:00010000:0.0:1788201326.490067:0:16421:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:0.0:1788201326.490070:0:16421:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff90235306d030, osc lock ffff902343fb7a20, flags 0x400041012 00010000:00010000:0.0:1788201326.490110:0:16421:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c516d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 16421 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.490120:0:16421:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c516d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 16421 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:0.0:1788201326.490136:0:16421:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372e40 (ffff902348a2fab8/0) at cl_lock_release():245 00000020:00010000:0.0:1788201326.490141:0:16421:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372e40 (ffff902348a2fab8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201326.490144:0:16421:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235306d030 (ffff902348a2f368/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201326.490148:0:16421:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372e40 (ffff902348a2fab8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201326.490151:0:16421:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235306d030 (ffff902348a2fab8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201326.490186:0:16421:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x407:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1788201326.490204:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 11 seconds 00010000:00010000:3.0:1788201326.490240:0:16420:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201326.490280:0:16417:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x409:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201326.490285:0:16417:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc6750 (ffff902350b9d5a8/0) at cl_lock_release():245 00000020:00010000:0.0:1788201326.490288:0:16417:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc6750 (ffff902350b9d5a8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201326.490292:0:16417:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398c30 (ffff902350b9dab8/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201326.490297:0:16417:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc6750 (ffff902350b9d5a8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201326.490299:0:16417:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398c30 (ffff902350b9d5a8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201326.490326:0:16417:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x409:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1788201326.490361:0:16420:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:3.0:1788201326.490378:0:16420:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5104 from it ffffa0b181bfbd78 00010000:00010000:3.0:1788201326.490383:0:16420:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201326.490390:0:16420:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201326.490419:0:16420:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201326.490429:0:16420:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00010000:3.0:1788201326.490435:0:16420:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5120 from it ffffa0b181bfbd78 00010000:00010000:3.0:1788201326.490439:0:16420:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201326.490446:0:16420:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201326.490462:0:16420:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201326.490536:0:16420:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00010000:3.0:1788201326.490544:0:16420:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5112 from it ffffa0b181bfbd78 00010000:00010000:3.0:1788201326.490547:0:16420:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201326.490560:0:16420:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201326.490592:0:16420:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5174 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 7 type: IBT flags: 0x10000000000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.490599:0:16420:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5174 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x0 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.490615:0:16420:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c517b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 8 type: IBT flags: 0x10000000000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.490621:0:16420:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c517b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x0 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.490636:0:16420:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c5182 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 9 type: IBT flags: 0x10000000000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.490642:0:16420:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c5182 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x0 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.490658:0:16420:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c5189 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 10 type: IBT flags: 0x10000000000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.490664:0:16420:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c5189 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x0 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.490681:0:16420:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5190 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 11 type: IBT flags: 0x10000000000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788201326.490702:0:16415:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x407:0x0](ffff9023449b4b08),name=file-5 00000002:00010000:2.0:1788201326.490721:0:16415:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x407:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201326.490731:0:16415:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201326.490741:0:16415:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201326.490749:0:16415:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201326.490758:0:16415:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x407:0x0](ffff9023449b4b08) 00000080:00010000:2.0:1788201326.490761:0:16415:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x407:0x0](ffff9023449b4b08) for lock 0x4605c7c97e1c5104 00000080:00010000:2.0:1788201326.490766:0:16415:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5104 from it ffffa0b181c23cb8 00010000:00010000:2.0:1788201326.490769:0:16415:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201326.490776:0:16415:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201326.490813:0:16415:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x407:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201326.490818:0:16415:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x407:0x0] 00000008:00010000:2.0:1788201326.490838:0:16415:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201326.490841:0:16415:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023533981b0, osc lock ffff9023500dd8f0, flags 0x400001002 00010000:00010000:2.0:1788201326.490850:0:16415:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902351d6bc00 type 11 mode 6 res 10737419264/1092 (0 18446744073709551615) 00010000:00010000:2.0:1788201326.490872:0:16415:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.490901:0:16415:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201326.490927:0:16415:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1788201326.490940:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201326.490962:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442704640:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201326.491007:0:16420:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5190 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x0 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201326.491040:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201326.491050:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442704768:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:2.0:1788201326.491067:0:16419:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201326.491078:0:16419:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00010000:2.0:1788201326.491084:0:16419:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5120 from it ffffa0b181c0bd78 00010000:00010000:2.0:1788201326.491087:0:16419:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201326.491095:0:16419:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201326.491114:0:16419:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201326.491121:0:16419:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00010000:2.0:1788201326.491127:0:16419:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5112 from it ffffa0b181c0bd78 00010000:00010000:2.0:1788201326.491131:0:16419:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201326.491137:0:16419:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201326.491161:0:16419:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c519e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 12 type: IBT flags: 0x10000000000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.491167:0:16419:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c519e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x0 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.491184:0:16419:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c51a5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 13 type: IBT flags: 0x10000000000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.491189:0:16419:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c51a5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x0 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.491203:0:16419:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c51ac lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 14 type: IBT flags: 0x10000000000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.491208:0:16419:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c51ac lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x0 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.491223:0:16419:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c51b3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 15 type: IBT flags: 0x10000000000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.491230:0:16419:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c51b3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT flags: 0x0 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.491248:0:16419:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c51ba lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 16 type: IBT flags: 0x10000000000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.491259:0:16419:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c51ba lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x0 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.491276:0:16419:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106c00/0x4605c7c97e1c51c1 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 17 type: IBT flags: 0x10000000000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.491283:0:16419:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106c00/0x4605c7c97e1c51c1 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT flags: 0x0 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201326.491317:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201326.491325:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442704896:192.168.201.130@tcp:63:ll_sa.0 00000100:00100000:1.0:1788201326.491562:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442704384:192.168.201.130@tcp:63:ll_sa.0 00000100:00100000:1.0:1788201326.491675:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201326.491683:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442704512:192.168.201.130@tcp:101:ll_sa.0 00000080:00200008:1.0:1788201326.491796:0:16413:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-3:[0x200000bd1:0x405:0x0](ffff9023449b5348) current size 0, valid attrs 41, mode 8180, uid 1204973568, gid -28637, new size 0, atime 1788201326.000000000, mtime 1788201326.000000000, ctime 1788201326.000000000 00010000:00010000:1.0:1788201326.491831:0:16413:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201326.491855:0:16413:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50b7 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4891 expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201326.491958:0:16413:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201326.491966:0:16413:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201326.491974:0:16413:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684200/0x4605c7c97e1c510b lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d49a9 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201326.491979:0:16417:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201326.492000:0:16417:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000020:00100000:1.0:1788201326.492011:0:16413:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 3 is allocated opc 36, max 7 00010000:00010000:0.0:1788201326.492016:0:16417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000100:00100000:1.0:1788201326.492024:0:16413:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458b8380 x1875064442705152/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1788201326.492038:0:16417:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:0.0:1788201326.492049:0:16417:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201326.492057:0:16417:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201326.492061:0:16417:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:0.0:1788201326.492067:0:16417:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181c33c08 00010000:00010000:0.0:1788201326.492071:0:16417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:0.0:1788201326.492079:0:16417:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:0.0:1788201326.492094:0:16417:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201326.492102:0:16417:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:0.0:1788201326.492109:0:16417:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201326.492113:0:16417:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-7, flags=9 00010000:00010000:0.0:1788201326.492178:0:16421:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c51c8 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 18 type: IBT flags: 0x10000000000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201326.492186:0:16413:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b8380 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16413:1875064442705152:192.168.201.130@tcp:36:chmod.0 00010000:00010000:0.0:1788201326.492202:0:16421:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c51c8 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT flags: 0x0 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201326.492247:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 11 seconds 00010000:00010000:1.0:1788201326.492521:0:16422:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c51cf lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 19 type: IBT flags: 0x10000000000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.492532:0:16422:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c51cf lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 19 type: IBT flags: 0x0 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.492563:0:16422:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201326.492572:0:16422:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00010000:1.0:1788201326.492581:0:16422:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5104 from it ffffa0b181c53d78 00010000:00010000:1.0:1788201326.492587:0:16422:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201326.492599:0:16422:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201326.492642:0:16422:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201326.492658:0:16422:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00010000:1.0:1788201326.492665:0:16422:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5120 from it ffffa0b181c53d78 00010000:00010000:1.0:1788201326.492670:0:16422:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201326.492677:0:16422:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201326.492707:0:16422:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201326.492717:0:16422:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00010000:1.0:1788201326.492724:0:16422:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5112 from it ffffa0b181c53d78 00010000:00010000:1.0:1788201326.492734:0:16422:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201326.492743:0:16422:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00000080:00200008:2.0:1788201326.492754:0:16417:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-7:[0x200000bd1:0x409:0x0](ffff9023449adb88) current size 0, valid attrs 41, mode 8180, uid 1117585408, gid -28637, new size 0, atime 1788201326.000000000, mtime 1788201326.000000000, ctime 1788201326.000000000 00010000:00010000:1.0:1788201326.492771:0:16422:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c51d6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 20 type: IBT flags: 0x10000000000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.492779:0:16417:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201326.492781:0:16422:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c51d6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x0 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.492830:0:16417:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c50ef lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4955 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201326.492838:0:16422:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c51dd lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 21 type: IBT flags: 0x10000000000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.492851:0:16422:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c51dd lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x0 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.492896:0:16417:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201326.492906:0:16417:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201326.492910:0:16417:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c5112 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d49e8 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00000020:00100000:2.0:1788201326.492927:0:16417:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 4 is allocated opc 36, max 7 00000100:00100000:2.0:1788201326.492938:0:16417:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c6d80 x1875064442705536/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201326.492959:0:16417:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6d80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16417:1875064442705536:192.168.201.130@tcp:36:chmod.0 00000100:00100000:2.0:1788201326.493038:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 11 seconds 00000100:00100000:0.0:1788201326.493159:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442705280:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:1.0:1788201326.493685:0:16422:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c51e4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 22 type: IBT flags: 0x10000000000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.493697:0:16422:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c51e4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT flags: 0x0 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.493732:0:16422:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c51eb lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 23 type: IBT flags: 0x10000000000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.493738:0:16422:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c51eb lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 23 type: IBT flags: 0x0 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201326.493821:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201326.493838:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442705664:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:1.0:1788201326.501779:0:2359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c516d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 16421 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.501809:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.501841:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x444:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.501862:0:2362:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.501911:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201326.501931:0:2362:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201326.501946:0:2362:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788201326.501957:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.501966:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.501976:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.501986:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201326.502013:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442704640:192.168.201.130@tcp:101:chmod.0 00010000:00010000:1.0:1788201326.502026:0:2359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c516d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a6d expref: -99 pid: 16421 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.502035:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502047:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5174 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 23 type: IBT flags: 0x0 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.502052:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502058:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5174 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 23 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.502063:0:2362:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502064:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5174 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 23 type: IBT flags: 0x10206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502072:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5174 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 23 type: IBT flags: 0x10206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.502074:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502081:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5174 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 23 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201326.502086:0:2362:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502088:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db0400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201326.502093:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5174 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 23 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201326.502095:0:2362:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1788201326.502101:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5174 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 23 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.502104:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502109:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5174 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 23 type: IBT flags: 0x20e400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.502119:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502122:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201326.502125:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502129:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5174 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 23 type: IBT flags: 0x4a0f400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.502132:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201326.502136:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5174 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 23 type: IBT flags: 0x4a0f400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201326.502143:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442703616:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201326.502159:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c517b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT flags: 0x0 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502165:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c517b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502171:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c517b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT flags: 0x10206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502177:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c517b lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT flags: 0x10206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502182:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c517b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502187:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db0800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201326.502189:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c517b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 22 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788201326.502195:0:16415:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x407:0x0] updating i_size 0 i_blocks 1 00010000:00010000:3.0:1788201326.502195:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c517b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 22 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1788201326.502201:0:16415:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc5720 (ffff902350b9de18/0) at cl_lock_release():245 00010000:00010000:3.0:1788201326.502201:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c517b lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 22 type: IBT flags: 0x20e400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1788201326.502204:0:16415:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc5720 (ffff902350b9de18/0) at cl_lock_cancel():142 00010000:00010000:3.0:1788201326.502207:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000020:00010000:2.0:1788201326.502210:0:16415:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023533981b0 (ffff902350b9d1b8/1) at cl_lock_cancel():142 00010000:00010000:3.0:1788201326.502210:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c517b lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 22 type: IBT flags: 0x4a0f400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1788201326.502215:0:16415:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc5720 (ffff902350b9de18/0) at cl_lock_fini():72 00010000:00010000:3.0:1788201326.502216:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c517b lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 22 type: IBT flags: 0x4a0f400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1788201326.502218:0:16415:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023533981b0 (ffff902350b9de18/1) at cl_lock_fini():72 00010000:00010000:3.0:1788201326.502228:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c5182 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x0 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502232:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c5182 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502237:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c5182 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x10206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502242:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c5182 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x10206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502247:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c5182 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788201326.502263:0:16415:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x407:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1788201326.502317:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db1a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201326.502321:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c5182 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 21 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502326:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c5182 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 21 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502332:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c5182 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 21 type: IBT flags: 0x20e400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502338:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201326.502341:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c5182 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 21 type: IBT flags: 0x4a0f400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1788201326.502346:0:16415:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00010000:00010000:3.0:1788201326.502347:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1a00/0x4605c7c97e1c5182 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 21 type: IBT flags: 0x4a0f400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502360:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c5189 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x0 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502375:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c5189 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502380:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c5189 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x10206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502386:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c5189 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x10206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502392:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c5189 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502398:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db0c00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201326.502401:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c5189 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 20 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502407:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c5189 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 20 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502414:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c5189 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 20 type: IBT flags: 0x20e400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502422:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201326.502425:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c5189 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 20 type: IBT flags: 0x4a0f400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502431:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0c00/0x4605c7c97e1c5189 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 20 type: IBT flags: 0x4a0f400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502444:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5190 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 19 type: IBT flags: 0x0 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502450:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5190 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 19 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201326.502452:0:16415:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201326.502455:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5190 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 19 type: IBT flags: 0x10206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502461:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5190 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 19 type: IBT flags: 0x10206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.502467:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5190 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 19 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.502472:0:16415:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:2.0:1788201326.502486:0:16415:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:2.0:1788201326.502501:0:16415:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201326.502509:0:16415:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201326.502514:0:16415:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:2.0:1788201326.502522:0:16415:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181c23c08 00010000:00010000:2.0:1788201326.502526:0:16415:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:2.0:1788201326.502534:0:16415:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:2.0:1788201326.502545:0:16415:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201326.502552:0:16415:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:2.0:1788201326.502560:0:16415:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201326.502564:0:16415:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-5, flags=9 00000080:00200008:2.0:1788201326.502587:0:16415:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-5:[0x200000bd1:0x407:0x0](ffff9023449b4b08) current size 0, valid attrs 41, mode 8180, uid 1204961280, gid -28637, new size 0, atime 1788201326.000000000, mtime 1788201326.000000000, ctime 1788201326.000000000 00010000:00010000:2.0:1788201326.502610:0:16415:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201326.502626:0:16415:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50d3 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d48f3 expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201326.502645:0:2359:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c516d lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4a6d expref: -99 pid: 16421 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.502672:0:16415:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788201326.502673:0:2359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c516d lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a6d expref: -99 pid: 16421 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.502707:0:16415:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201326.502720:0:16415:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5104 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d49cc expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000020:00100000:2.0:1788201326.502741:0:16415:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 5 is allocated opc 36, max 7 00000100:00100000:2.0:1788201326.502753:0:16415:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c5880 x1875064442705920/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000008:00010000:1.0:1788201326.502757:0:2359:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c516d lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a6d expref: -99 pid: 16421 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201326.502791:0:16415:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16415:1875064442705920:192.168.201.130@tcp:36:chmod.0 00010000:00010000:1.0:1788201326.502805:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c516d lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a6d expref: -99 pid: 16421 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201326.502929:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 11 seconds 00010000:00010000:1.0:1788201326.503834:0:2359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c516d lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a6d expref: -99 pid: 16421 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.503859:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.503860:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c516d lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a6d expref: -99 pid: 16421 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.503943:0:2359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c516d lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4a6d expref: -99 pid: 16421 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.503945:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.503987:0:2360:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201326.504000:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442704512:192.168.201.130@tcp:101:ll_sa.0 00010000:00010000:0.0:1788201326.504003:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788201326.504023:0:2360:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788201326.504036:0:2360:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1788201326.504050:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.504064:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.504070:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.504078:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788201326.504090:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442703744:192.168.201.130@tcp:101:chmod.0 00000080:00200000:0.0:1788201326.504135:0:16411:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x403:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201326.504139:0:16411:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235844cde0 (ffff9023500e7638/0) at cl_lock_release():245 00000020:00010000:0.0:1788201326.504142:0:16411:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235844cde0 (ffff9023500e7638/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201326.504146:0:16411:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023450454b0 (ffff9023500e7128/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201326.504150:0:16411:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235844cde0 (ffff9023500e7638/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201326.504152:0:16411:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023450454b0 (ffff9023500e7638/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201326.504191:0:16411:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x403:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201326.504255:0:16411:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201326.504271:0:16411:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201326.504285:0:16411:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:0.0:1788201326.504296:0:16411:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:0.0:1788201326.504309:0:16411:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201326.504316:0:16411:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201326.504320:0:16411:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:0.0:1788201326.504327:0:16411:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181c03c08 00010000:00010000:0.0:1788201326.504329:0:16411:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:0.0:1788201326.504335:0:16411:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:0.0:1788201326.504343:0:16411:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201326.504347:0:16411:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:0.0:1788201326.504352:0:16411:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201326.504356:0:16411:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-1, flags=9 00010000:00010000:3.0:1788201326.507136:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db0a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201326.507145:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5190 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 19 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.507154:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5190 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 19 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.507164:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5190 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 19 type: IBT flags: 0x20e400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.507173:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201326.507184:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5190 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 19 type: IBT flags: 0x4a0f400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201326.507191:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c5190 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 19 type: IBT flags: 0x4a0f400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201326.507234:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442704768:192.168.201.130@tcp:63:ll_sa.0 00000080:00200000:2.0:1788201326.509060:0:16416:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x408:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201326.509076:0:16416:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc69a0 (ffff902350b9d248/0) at cl_lock_release():245 00000020:00010000:2.0:1788201326.509082:0:16416:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc69a0 (ffff902350b9d248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201326.509088:0:16416:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398ab0 (ffff902350b9d7e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201326.509095:0:16416:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc69a0 (ffff902350b9d248/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201326.509098:0:16416:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398ab0 (ffff902350b9d248/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201326.509150:0:16416:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x408:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201326.509236:0:16416:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201326.509268:0:16416:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201326.509294:0:16416:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:2.0:1788201326.509312:0:16416:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:2.0:1788201326.509322:0:16416:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201326.509332:0:16416:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201326.509336:0:16416:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:2.0:1788201326.509344:0:16416:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181c2bc08 00010000:00010000:2.0:1788201326.510550:0:16416:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:2.0:1788201326.510578:0:16416:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:2.0:1788201326.510605:0:16416:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201326.510619:0:16416:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=25 00000080:00200000:2.0:1788201326.510629:0:16416:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201326.510634:0:16416:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-6, flags=9 00000080:00200008:2.0:1788201326.510662:0:16416:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-6:[0x200000bd1:0x408:0x0](ffff9023449ab248) current size 0, valid attrs 41, mode 8180, uid 1204961280, gid -28637, new size 0, atime 1788201326.000000000, mtime 1788201326.000000000, ctime 1788201326.000000000 00010000:00010000:2.0:1788201326.510694:0:16416:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201326.510718:0:16416:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c50e1 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4924 expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201326.510776:0:16416:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201326.510784:0:16416:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201326.510787:0:16416:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684800/0x4605c7c97e1c5120 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4a20 expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00000020:00100000:2.0:1788201326.510814:0:16416:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 6 is allocated opc 36, max 7 00000100:00100000:2.0:1788201326.510830:0:16416:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458c4e00 x1875064442706176/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201326.510859:0:16416:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4e00 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16416:1875064442706176:192.168.201.130@tcp:36:chmod.0 00000100:00100000:2.0:1788201326.510991:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 11 seconds 00000080:00200008:0.0:1788201326.511176:0:16411:0:(llite_lib.c:2436:ll_setattr_raw()) START file file-1:[0x200000bd1:0x403:0x0](ffff90234497ec08) current size 0, valid attrs 41, mode 8180, uid 1117601792, gid -28637, new size 0, atime 1788201326.000000000, mtime 1788201326.000000000, ctime 1788201326.000000000 00010000:00010000:0.0:1788201326.511194:0:16411:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201326.511208:0:16411:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c509b lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d482f expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201326.511232:0:16411:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201326.511239:0:16411:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201326.511242:0:16411:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c5119 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4a04 expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201326.511269:0:16411:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 7 is allocated opc 36, max 7 00000100:00100000:0.0:1788201326.511276:0:16411:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b3480 x1875064442706432/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201326.511291:0:16411:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b3480 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16411:1875064442706432:192.168.201.130@tcp:36:chmod.0 00000100:00100000:0.0:1788201326.511324:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 11 seconds 00010000:00010000:0.0:1788201326.511414:0:16427:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c51f2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 19 type: IBT flags: 0x10000000000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.511424:0:16427:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c51f2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 19 type: IBT flags: 0x0 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.511455:0:16427:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c51f9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 20 type: IBT flags: 0x10000000000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.511461:0:16427:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c51f9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x0 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.511482:0:16427:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5200 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 21 type: IBT flags: 0x10000000000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.511489:0:16427:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5200 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x0 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.511508:0:16427:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c5207 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 22 type: IBT flags: 0x10000000000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.511515:0:16427:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c5207 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT flags: 0x0 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.511538:0:16427:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c520e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 23 type: IBT flags: 0x10000000000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.511544:0:16427:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c520e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 23 type: IBT flags: 0x0 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.511560:0:16427:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c5215 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 24 type: IBT flags: 0x10000000000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.511566:0:16427:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c5215 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x0 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.511582:0:16427:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c521c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 25 type: IBT flags: 0x10000000000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.511588:0:16427:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c521c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x0 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.511607:0:16427:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5223 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 26 type: IBT flags: 0x10000000000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.511613:0:16427:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5223 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 26 type: IBT flags: 0x0 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201326.515450:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201326.515488:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442706560:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:2.0:1788201326.515838:0:2362:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c519e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 26 type: IBT flags: 0x0 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.515852:0:2362:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c519e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 26 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.515860:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c519e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 26 type: IBT flags: 0x10206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.515867:0:2362:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c519e lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 26 type: IBT flags: 0x10206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.515893:0:2362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c519e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 26 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.515901:0:2362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351685200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201326.515906:0:2362:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c519e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 26 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.515913:0:2362:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c519e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 26 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.515921:0:2362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c519e lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 26 type: IBT flags: 0x20e400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.515930:0:2362:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201326.515938:0:2362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c519e lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 26 type: IBT flags: 0x4a0f400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.515945:0:2362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c519e lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 26 type: IBT flags: 0x4a0f400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.515972:0:2362:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c51a5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x0 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.515980:0:2362:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c51a5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.515987:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c51a5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x10206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.515994:0:2362:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c51a5 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x10206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516001:0:2362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c51a5 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516006:0:2362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351684400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201326.516010:0:2362:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c51a5 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 25 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516016:0:2362:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c51a5 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 25 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516025:0:2362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c51a5 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 25 type: IBT flags: 0x20e400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516031:0:2362:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201326.516035:0:2362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c51a5 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 25 type: IBT flags: 0x4a0f400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516041:0:2362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c51a5 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 25 type: IBT flags: 0x4a0f400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516057:0:2362:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c51ac lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x0 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516065:0:2362:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c51ac lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.516068:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c51c8 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x0 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516072:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c51ac lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x10206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516104:0:2362:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c51ac lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x10206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516114:0:2362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c51ac lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516121:0:2362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351685600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201326.516123:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c51c8 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516126:0:2362:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c51ac lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 25 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516151:0:2362:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c51ac lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 25 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.516175:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c51c8 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x10206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.516186:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c51c8 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x10206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516197:0:2362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c51ac lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 25 type: IBT flags: 0x20e400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.516198:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c51c8 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516233:0:2362:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201326.516236:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839d800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201326.516240:0:2362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c51ac lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 24 type: IBT flags: 0x4a0f400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.516245:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c51c8 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 25 type: IBT flags: 0x206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516258:0:2362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685600/0x4605c7c97e1c51ac lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 25 type: IBT flags: 0x4a0f400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.516263:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c51c8 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 25 type: IBT flags: 0x206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.516277:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c51c8 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 23 type: IBT flags: 0x20e400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516283:0:2362:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c51b3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x0 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.516285:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201326.516296:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c51c8 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 23 type: IBT flags: 0x4a0f400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516296:0:2362:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c51b3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516326:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c51b3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x10206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516334:0:2362:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c51b3 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x10206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.516345:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c51c8 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 24 type: IBT flags: 0x4a0f400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201326.516391:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442705280:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:0.0:1788201326.516475:0:2359:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5151 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 23 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4aba expref: -99 pid: 16421 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.516494:0:2359:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5151 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4aba expref: -99 pid: 16421 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.516503:0:2359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5151 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4aba expref: -99 pid: 16421 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.516515:0:2359:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5151 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4aba expref: -99 pid: 16421 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.516583:0:2359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5151 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4aba expref: -99 pid: 16421 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.516597:0:2359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5151 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4aba expref: -99 pid: 16421 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.516604:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5151 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4aba expref: -99 pid: 16421 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.516611:0:2359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5151 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4aba expref: -99 pid: 16421 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201326.516627:0:2359:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023488b1880 x1875064442704384/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 1840/4320 e 0 to 0 dl 1788201342 ref 1 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:0.0:1788201326.516639:0:2359:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5151 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4aba expref: -99 pid: 16421 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:0.0:1788201326.516653:0:2359:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5151 from it ffff9023436b6990 00010000:00010000:0.0:1788201326.516659:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5151 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4aba expref: -99 pid: 16421 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.516674:0:2359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5151 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4aba expref: -99 pid: 16421 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.516706:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5158 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT flags: 0x0 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516804:0:2362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c51b3 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516814:0:2362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351684e00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201326.516818:0:2362:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c51b3 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 22 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516825:0:2362:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c51b3 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 22 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516832:0:2362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c51b3 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 22 type: IBT flags: 0x20e400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516839:0:2362:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201326.516843:0:2362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c51b3 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 22 type: IBT flags: 0x4a0f400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516850:0:2362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684e00/0x4605c7c97e1c51b3 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 22 type: IBT flags: 0x4a0f400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516867:0:2362:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c51ba lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x0 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516920:0:2362:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c51ba lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516928:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c51ba lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x10206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516934:0:2362:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c51ba lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x10206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516940:0:2362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c51ba lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516945:0:2362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345106200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201326.516948:0:2362:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c51ba lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 21 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516954:0:2362:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c51ba lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 21 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516959:0:2362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c51ba lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 21 type: IBT flags: 0x20e400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516964:0:2362:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201326.516967:0:2362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c51ba lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 21 type: IBT flags: 0x4a0f400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516973:0:2362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c51ba lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 21 type: IBT flags: 0x4a0f400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516985:0:2362:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106c00/0x4605c7c97e1c51c1 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x0 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516991:0:2362:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106c00/0x4605c7c97e1c51c1 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.516997:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106c00/0x4605c7c97e1c51c1 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x10206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.517003:0:2362:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106c00/0x4605c7c97e1c51c1 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x10206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.517009:0:2362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106c00/0x4605c7c97e1c51c1 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.517014:0:2362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345106c00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201326.517017:0:2362:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106c00/0x4605c7c97e1c51c1 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 20 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.517023:0:2362:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106c00/0x4605c7c97e1c51c1 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 20 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.517029:0:2362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106c00/0x4605c7c97e1c51c1 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 20 type: IBT flags: 0x20e400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.517034:0:2362:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201326.517037:0:2362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106c00/0x4605c7c97e1c51c1 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 20 type: IBT flags: 0x4a0f400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201326.517044:0:2362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106c00/0x4605c7c97e1c51c1 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 20 type: IBT flags: 0x4a0f400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201326.517061:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442704896:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:0.0:1788201326.522546:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5158 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT flags: 0x206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522556:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5158 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT flags: 0x10206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522564:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5158 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT flags: 0x10206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522572:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5158 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT flags: 0x206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522581:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839d000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201326.522586:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5158 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 18 type: IBT flags: 0x206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522593:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5158 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 18 type: IBT flags: 0x206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522602:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5158 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 18 type: IBT flags: 0x20e400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522611:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201326.522618:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5158 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 18 type: IBT flags: 0x4a0f400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522625:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5158 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 18 type: IBT flags: 0x4a0f400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522657:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c515f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT flags: 0x0 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522662:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c515f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT flags: 0x206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522668:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c515f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT flags: 0x10206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522673:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c515f lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT flags: 0x10206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522677:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c515f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT flags: 0x206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522681:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839c800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201326.522684:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c515f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 17 type: IBT flags: 0x206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522688:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c515f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 17 type: IBT flags: 0x206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522693:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c515f lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 17 type: IBT flags: 0x20e400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522697:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201326.522699:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c515f lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 17 type: IBT flags: 0x4a0f400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522705:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c515f lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 17 type: IBT flags: 0x4a0f400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522716:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c5166 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x0 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522721:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c5166 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522727:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c5166 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x10206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522733:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c5166 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x10206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522739:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c5166 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522744:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839c600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201326.522746:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c5166 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 16 type: IBT flags: 0x206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522751:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c5166 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 16 type: IBT flags: 0x206400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522755:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c5166 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 16 type: IBT flags: 0x20e400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522759:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201326.522762:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c5166 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 16 type: IBT flags: 0x4a0f400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.522768:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c5166 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 16 type: IBT flags: 0x4a0f400000000 pid: 16421 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201326.522790:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442704384:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:0.0:1788201326.523087:0:2359:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c51cf lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b00 expref: -99 pid: 16422 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.523104:0:2359:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c51cf lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b00 expref: -99 pid: 16422 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.523114:0:2359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c51cf lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b00 expref: -99 pid: 16422 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.523125:0:2359:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c51cf lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b00 expref: -99 pid: 16422 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.523135:0:2359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c51cf lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b00 expref: -99 pid: 16422 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.523144:0:2359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c51cf lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b00 expref: -99 pid: 16422 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.523151:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c51cf lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b00 expref: -99 pid: 16422 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.523158:0:2359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c51cf lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b00 expref: -99 pid: 16422 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201326.523175:0:2359:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458b9f80 x1875064442705664/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 2232/5072 e 0 to 0 dl 1788201342 ref 1 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:0.0:1788201326.523187:0:2359:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c51cf lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b00 expref: -99 pid: 16422 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.523256:0:11147:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5151 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4aba expref: -99 pid: 16421 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201326.523390:0:11147:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x406:0x0](ffff902344979148): rc = 0 00000080:00200000:0.0:1788201326.523414:0:11147:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x406:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201326.524370:0:11147:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x406:0x0](ffff902344979148) for lock 0x4605c7c97e1c5151 00000080:00010000:0.0:1788201326.524386:0:11147:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5151 from it ffff9023436b6990 00010000:00010000:0.0:1788201326.524391:0:11147:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5151 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4aba expref: -99 pid: 16421 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.524403:0:11147:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5151 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4aba expref: -99 pid: 16421 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201326.526156:0:2360:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c51f2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b38 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526185:0:2360:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c51f2 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b38 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526194:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c51f2 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b38 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526214:0:2360:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c51f2 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b38 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526226:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c51f2 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b38 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526240:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c51f2 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b38 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526248:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c51f2 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b38 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526256:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c51f2 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b38 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201326.526276:0:2360:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023488b3100 x1875064442706560/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3408/7976 e 0 to 0 dl 1788201342 ref 1 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:0.0:1788201326.526289:0:2360:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c51f2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b38 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:0.0:1788201326.526306:0:2360:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c51f2 from it ffff90235262c590 00010000:00010000:0.0:1788201326.526310:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c51f2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b38 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526317:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c51f2 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b38 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.526345:0:2360:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c51f9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b54 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526354:0:2360:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c51f9 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b54 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526360:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c51f9 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b54 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526378:0:2360:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c51f9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b54 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526385:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c51f9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b54 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526394:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c51f9 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b54 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526402:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c51f9 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b54 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526410:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c51f9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b54 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201326.526420:0:2360:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023488b3100 x1875064442706560/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3408/7976 e 0 to 0 dl 1788201342 ref 2 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:0.0:1788201326.526431:0:2360:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c51f9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b54 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:0.0:1788201326.526439:0:2360:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c51f9 from it ffff90235262d990 00010000:00010000:0.0:1788201326.526441:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c51f9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b54 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526446:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c51f9 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b54 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.526456:0:2360:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5200 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b70 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526462:0:2360:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5200 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b70 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526466:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5200 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b70 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526474:0:2360:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5200 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b70 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526482:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5200 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b70 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526489:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5200 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b70 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526502:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5200 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b70 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526509:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5200 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b70 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201326.526519:0:2360:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023488b3100 x1875064442706560/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3408/7976 e 0 to 0 dl 1788201342 ref 3 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:0.0:1788201326.526526:0:2360:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5200 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b70 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:0.0:1788201326.526533:0:2360:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5200 from it ffff90235262c190 00010000:00010000:0.0:1788201326.526535:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5200 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b70 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526540:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5200 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b70 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.526551:0:2360:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c5207 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b8c expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526556:0:2360:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c5207 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b8c expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526561:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c5207 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b8c expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526570:0:2360:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c5207 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b8c expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526575:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c5207 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b8c expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526579:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c5207 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b8c expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526583:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c5207 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b8c expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526586:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c5207 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b8c expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201326.526592:0:2360:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023488b3100 x1875064442706560/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3408/7976 e 0 to 0 dl 1788201342 ref 4 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:0.0:1788201326.526597:0:2360:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c5207 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b8c expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:0.0:1788201326.526601:0:2360:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5207 from it ffff90235262f190 00010000:00010000:0.0:1788201326.526603:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c5207 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b8c expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526606:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c5207 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b8c expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.526618:0:2360:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c520e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ba8 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526624:0:2360:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c520e lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ba8 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526628:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c520e lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ba8 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526632:0:2360:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c520e lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4ba8 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526636:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c520e lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ba8 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526640:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c520e lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4ba8 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526644:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c520e lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4ba8 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526647:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c520e lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4ba8 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201326.526651:0:2360:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023488b3100 x1875064442706560/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3408/7976 e 0 to 0 dl 1788201342 ref 5 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:0.0:1788201326.526660:0:2360:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c520e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ba8 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:0.0:1788201326.526665:0:2360:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c520e from it ffff90235262d190 00010000:00010000:0.0:1788201326.526666:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c520e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4ba8 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526670:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c520e lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ba8 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.526678:0:2360:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c5215 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4bc4 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526682:0:2360:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c5215 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4bc4 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526686:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c5215 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4bc4 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526690:0:2360:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c5215 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4bc4 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526694:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c5215 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4bc4 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526699:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c5215 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4bc4 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526706:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c5215 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4bc4 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526710:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c5215 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4bc4 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201326.526717:0:2360:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023488b3100 x1875064442706560/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3408/7976 e 0 to 0 dl 1788201342 ref 6 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:0.0:1788201326.526724:0:2360:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c5215 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4bc4 expref: -99 pid: 16427 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:0.0:1788201326.526732:0:2360:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5215 from it ffff90235262e990 00010000:00010000:0.0:1788201326.526735:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c5215 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4bc4 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526743:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c5215 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4bc4 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.526755:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c521c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x0 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526763:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c521c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526769:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c521c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x10206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526779:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c521c lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x10206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526784:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c521c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526790:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839c000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201326.526794:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c521c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526798:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c521c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526803:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c521c lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x20e400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526807:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201326.526812:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c521c lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x4a0f400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526815:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c521c lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x4a0f400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526834:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5223 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x0 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526838:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5223 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526842:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5223 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x10206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526846:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5223 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x10206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526850:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5223 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526853:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902343730e00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201326.526854:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5223 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526857:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5223 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526860:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5223 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x20e400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526863:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201326.526864:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5223 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x4a0f400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201326.526867:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5223 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x4a0f400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201326.526906:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442706560:192.168.201.130@tcp:63:ll_sa.0 00000080:00010000:1.0:1788201326.527255:0:2359:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c51cf from it ffff902352604990 00010000:00010000:1.0:1788201326.527263:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c51cf lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b00 expref: -99 pid: 16422 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527273:0:2359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c51cf lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4b00 expref: -99 pid: 16422 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201326.527294:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c51d6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527300:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c51d6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527305:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c51d6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527310:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c51d6 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527316:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c51d6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527321:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345802000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201326.527324:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c51d6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527330:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c51d6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527336:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c51d6 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x20e400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527342:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201326.527345:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c51d6 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527352:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802000/0x4605c7c97e1c51d6 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527376:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c51dd lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527383:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c51dd lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527389:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c51dd lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527395:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c51dd lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527401:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c51dd lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527407:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345802400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201326.527410:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c51dd lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527415:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c51dd lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527421:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c51dd lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x20e400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527426:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201326.527430:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c51dd lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527436:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c51dd lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527448:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c51e4 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527455:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c51e4 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527461:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c51e4 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527467:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c51e4 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527473:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c51e4 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527479:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345803200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201326.527482:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c51e4 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527488:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c51e4 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527494:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c51e4 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x20e400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527499:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201326.527503:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c51e4 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527509:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c51e4 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527521:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c51eb lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527527:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c51eb lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527532:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c51eb lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527538:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c51eb lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527544:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c51eb lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527549:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345803800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201326.527552:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c51eb lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527557:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c51eb lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527562:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c51eb lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x20e400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527567:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201326.527570:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c51eb lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201326.527576:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c51eb lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201326.527590:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442705664:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:1.0:1788201326.527625:0:21:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c51cf lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b00 expref: -99 pid: 16422 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.527792:0:11147:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c51f2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b38 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201326.527851:0:11147:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x404:0x0](ffff9023449a7448): rc = 0 00000080:00200000:0.0:1788201326.527869:0:11147:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x404:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201326.527901:0:11147:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x404:0x0](ffff9023449a7448) for lock 0x4605c7c97e1c51f2 00000080:00010000:0.0:1788201326.528773:0:11147:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c51f2 from it ffff90235262c590 00010000:00010000:0.0:1788201326.528784:0:11147:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c51f2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4b38 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.528795:0:11147:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c51f2 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4b38 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201326.528828:0:11147:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c51f9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b54 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201326.529895:0:11147:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x405:0x0](ffff9023449b5348): rc = 0 00000080:00200000:0.0:1788201326.530017:0:11147:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x405:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201326.530034:0:11147:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x405:0x0](ffff9023449b5348) for lock 0x4605c7c97e1c51f9 00000080:00010000:0.0:1788201326.530043:0:11147:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c51f9 from it ffff90235262d990 00010000:00010000:0.0:1788201326.530048:0:11147:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c51f9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4b54 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.530062:0:11147:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c51f9 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4b54 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201326.530095:0:11147:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5200 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b70 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201326.530129:0:11147:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x406:0x0](ffff902344979148): rc = 0 00000080:00200000:0.0:1788201326.530141:0:11147:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x406:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201326.530149:0:11147:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x406:0x0](ffff902344979148) for lock 0x4605c7c97e1c5200 00000080:00010000:0.0:1788201326.530153:0:11147:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5200 from it ffff90235262c190 00010000:00010000:0.0:1788201326.530157:0:11147:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5200 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4b70 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.530164:0:11147:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5200 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4b70 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201326.530181:0:11147:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c5207 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4b8c expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201326.530208:0:11147:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x407:0x0](ffff9023449b4b08): rc = 0 00000080:00200000:0.0:1788201326.530220:0:11147:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x407:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201326.530227:0:11147:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x407:0x0](ffff9023449b4b08) for lock 0x4605c7c97e1c5207 00000080:00010000:0.0:1788201326.530231:0:11147:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5207 from it ffff90235262f190 00010000:00010000:0.0:1788201326.530235:0:11147:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c5207 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4b8c expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.530243:0:11147:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c5207 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4b8c expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201326.530260:0:11147:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c520e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4ba8 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201326.530289:0:11147:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x408:0x0](ffff9023449ab248): rc = 0 00000080:00200000:0.0:1788201326.530301:0:11147:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x408:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201326.530308:0:11147:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x408:0x0](ffff9023449ab248) for lock 0x4605c7c97e1c520e 00000080:00010000:0.0:1788201326.530312:0:11147:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c520e from it ffff90235262d190 00010000:00010000:0.0:1788201326.530315:0:11147:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c520e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4ba8 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.530328:0:11147:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c520e lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4ba8 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201326.530346:0:11147:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c5215 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4bc4 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201326.530384:0:11147:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x409:0x0](ffff9023449adb88): rc = 0 00000080:00200000:0.0:1788201326.530395:0:11147:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x409:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201326.530402:0:11147:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x409:0x0](ffff9023449adb88) for lock 0x4605c7c97e1c5215 00000080:00010000:0.0:1788201326.530406:0:11147:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5215 from it ffff90235262e990 00010000:00010000:0.0:1788201326.530408:0:11147:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c5215 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4bc4 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201326.530415:0:11147:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c5215 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4bc4 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201326.530639:0:21:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x406:0x0](ffff902344979148): rc = 0 00000080:00200000:1.0:1788201326.530671:0:21:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x406:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1788201326.530682:0:21:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x406:0x0](ffff902344979148) for lock 0x4605c7c97e1c51cf 00000080:00010000:1.0:1788201326.530689:0:21:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c51cf from it ffff902352604990 00010000:00010000:1.0:1788201326.530692:0:21:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c51cf lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4b00 expref: -99 pid: 16422 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201326.530701:0:21:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c51cf lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4b00 expref: -99 pid: 16422 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201327.234069:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201328.263576:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201329.281032:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201330.305170:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201330.563522:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442706688:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201330.563625:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442706944:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201330.563670:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201330.563677:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442706816:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201330.563701:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442707072:192.168.201.130@tcp:400:kworker.0 00000100:00080000:3.0:1788201330.566621:0:2362:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff902350a11000 of lustre-MDT0000_UUID@192.168.201.130@tcp abruptly disconnected: reconnecting 00000100:00080000:3.0:1788201330.566629:0:2362:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1788201330.566634:0:2362:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff902350a11000: Connection to lustre-MDT0000 (at 192.168.201.130@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1788201330.581377:0:2362:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201330.581393:0:2362:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 575 00000100:00080000:3.0:1788201330.582841:0:2362:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201330.582856:0:2362:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201330.583098:0:2362:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442707200/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201330.583142:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442706816:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201330.583176:0:2362:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff902350a11000: committing for last_committed 4294968868 gen 2 00000100:00100000:3.0:1788201330.583205:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442707072:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201330.583218:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442706688:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201330.583229:0:2362:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0000-osc-ffff902350a11000: committing for last_committed 4294968868 gen 2 00000100:00100000:3.0:1788201330.583243:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442706944:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201330.583471:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442707200:192.168.201.130@tcp:38:ptlrpcd_01_01.0 00000100:00080000:0.0:1788201330.590808:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201330.590829:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-19) 00000100:00100000:0.0:1788201330.590835:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442707200:192.168.201.130@tcp:38:ptlrpcd_01_01.0 00010000:00010000:1.0:1788201331.329778:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201332.354806:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201333.377222:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201334.401308:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201335.425071:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201335.681190:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201335.681196:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201335.681200:0:37:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201335.681229:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 607 00000100:00080000:3.0:1788201335.681245:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788201335.681255:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201335.681268:0:37:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201335.681287:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442707456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201335.681360:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442707456:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201335.681556:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201335.681574:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442707328:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201335.681843:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442707712:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201335.681895:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201335.681899:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442707584:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201335.695238:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442707584:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201335.695267:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442707712:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201336.450060:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201337.473782:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201337.729055:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 1 seconds 00000100:00100000:2.0:1788201337.729055:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 1 seconds 00000100:00100000:0.0:1788201337.731564:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 1 seconds 00000100:00100000:3.0:1788201337.731757:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 1 seconds 00000100:00100000:0.0:1788201337.731850:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 1 seconds 00000100:00100000:2.0:1788201337.731850:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 1 seconds 00000100:00100000:1.0:1788201337.731913:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 1 seconds 00010000:00010000:1.0:1788201338.503265:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201338.753082:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 1 seconds 00000100:00100000:2.0:1788201338.753646:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 1 seconds 00000100:00100000:0.0:1788201338.753697:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 1 seconds 00000100:00100000:3.0:1788201338.755446:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 1 seconds 00000100:00100000:2.0:1788201338.755448:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 1 seconds 00000100:00100000:0.0:1788201338.756260:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 1 seconds 00000100:00100000:2.0:1788201338.756451:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 1 seconds 00010000:00010000:1.0:1788201339.521199:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201339.777050:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 1 seconds 00000100:00100000:3.0:1788201339.779960:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 1 seconds 00000100:00100000:3.0:1788201339.779999:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 1 seconds 00000100:00100000:1.0:1788201339.781990:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 1 seconds 00000100:00100000:2.0:1788201339.782507:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 1 seconds 00000100:00100000:3.0:1788201339.782738:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 1 seconds 00000100:00100000:0.0:1788201339.782917:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 1 seconds 00010000:00010000:2.0:1788201340.552288:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201340.804760:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 1 seconds 00000100:00100000:3.0:1788201340.804784:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 1 seconds 00000100:00080000:3.0:1788201340.804892:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201340.804898:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1788201340.804970:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442707840:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201340.805109:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442708096:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201340.805182:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201340.805189:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442707968:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201340.806124:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 1 seconds 00000100:00100000:2.0:1788201340.806295:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 1 seconds 00000100:00100000:0.0:1788201340.806408:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 1 seconds 00000100:00100000:1.0:1788201340.806507:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 1 seconds 00000100:00100000:2.0:1788201340.806650:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 1 seconds 00000100:00100000:3.0:1788201340.812715:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442707968:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201340.812740:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442708096:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201341.569595:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201341.825013:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 1 seconds 00000100:00100000:1.0:1788201341.827124:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 1 seconds 00000100:00100000:0.0:1788201341.830305:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 1 seconds 00000100:00100000:1.0:1788201341.830382:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 1 seconds 00000100:00100000:3.0:1788201341.831491:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 1 seconds 00000100:00100000:1.0:1788201341.832026:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 1 seconds 00000100:00100000:0.0:1788201341.833000:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 1 seconds 00010000:00010000:1.0:1788201342.599018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1788201342.849136:0:16413:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201326/real 1788201326] req@ffff9023458b8380 x1875064442705152/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201342 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1788201342.852492:0:16412:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201326/real 1788201326] req@ffff9023488b1180 x1875064442704000/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201342 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201342.852539:0:16412:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 8, was 6): CONNECTING 00000100:00100000:0.0:1788201342.852559:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00000400:0.0:1788201342.852593:0:16414:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201326/real 1788201326] req@ffff902352689500 x1875064442704256/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201342 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201342.852607:0:16414:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 8, was 6): CONNECTING 00000100:00100000:0.0:1788201342.852610:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00000400:0.0:1788201342.852650:0:16415:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201326/real 1788201326] req@ffff9023458c5880 x1875064442705920/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201342 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201342.852663:0:16415:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 8, was 6): CONNECTING 00000100:00100000:0.0:1788201342.852666:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00000400:2.0:1788201342.853032:0:16411:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201326/real 1788201326] req@ffff9023488b3480 x1875064442706432/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201342 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201342.853049:0:16411:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 8, was 6): CONNECTING 00000100:00100000:2.0:1788201342.853052:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00000400:3.0:1788201342.856325:0:16416:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201326/real 1788201326] req@ffff9023458c4e00 x1875064442706176/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201342 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201342.856365:0:16416:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 8, was 6): CONNECTING 00000100:00100000:3.0:1788201342.856372:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00000400:0.0:1788201342.857031:0:16417:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201326/real 1788201326] req@ffff9023458c6d80 x1875064442705536/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201342 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201342.857051:0:16417:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 8, was 6): CONNECTING 00000100:00100000:0.0:1788201342.857055:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00080000:1.0:1788201342.908324:0:16413:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 8, was 6): CONNECTING 00000100:00100000:1.0:1788201342.908341:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:2.0:1788201343.619128:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201343.873015:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:0.0:1788201343.873046:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:0.0:1788201343.873059:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788201343.873164:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201343.875463:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788201343.876783:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:1.0:1788201343.938442:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:2.0:1788201344.641191:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201344.897200:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:3.0:1788201344.897457:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788201344.905310:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:2.0:1788201344.905325:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788201344.905411:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201344.905470:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:1.0:1788201344.961006:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:0.0:1788201345.665203:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201345.921286:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:1.0:1788201345.921405:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:2.0:1788201345.921795:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:0.0:1788201345.921800:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788201345.923761:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:1.0:1788201345.923863:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788201345.923948:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201345.923956:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1788201345.924147:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442708352:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201345.924596:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442708224:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201345.924650:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442708480:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201345.932947:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442708352:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201345.933194:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442708480:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201345.985196:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:1.0:1788201346.694274:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201346.945042:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788201346.945042:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:1.0:1788201346.945096:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:1.0:1788201346.945115:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:1.0:1788201346.945141:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788201346.946339:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:1.0:1788201347.008979:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:0.0:1788201347.725257:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201347.969162:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:1.0:1788201347.969371:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:1.0:1788201347.969384:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788201347.969450:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:0.0:1788201347.969604:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788201347.977243:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:1.0:1788201348.033006:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:2.0:1788201348.737117:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201348.993127:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788201348.993127:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:1.0:1788201348.994617:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:2.0:1788201348.994679:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:3.0:1788201348.994728:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788201349.001632:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:1.0:1788201349.057144:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:0.0:1788201349.761920:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201350.017062:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201350.017065:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788201350.017116:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201335/real 1788201335] req@ffff90235268a300 x1875064442707456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201350 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201350.017129:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:0.0:1788201350.017159:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442707456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201350 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201350.017202:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201350.022908:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:3.0:1788201350.024539:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788201350.027893:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201350.027933:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201350.027941:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442707456:192.168.201.130@tcp:38:kworker.0 00000100:00080000:3.0:1788201350.028212:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201350.028216:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201350.028219:0:37:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201350.028225:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 612 00000100:00080000:3.0:1788201350.028235:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1788201350.028242:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201350.028253:0:37:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201350.028271:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442708736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201350.028332:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442708864:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201350.028438:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201350.028446:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442708608:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201350.028476:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442708992:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201350.029074:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442708736:192.168.201.130@tcp:38:kworker.0 00000100:00080000:2.0:1788201350.035663:0:2362:0:(recover.c:205:ptlrpc_request_handle_notconn()) import MGC192.168.201.130@tcp of MGS@192.168.201.130@tcp abruptly disconnected: reconnecting 00000100:00080000:2.0:1788201350.035674:0:2362:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from FULL to DISCONN 00000100:02020000:2.0:1788201350.035680:0:2362:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail 00000100:00080000:2.0:1788201350.048765:0:2362:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.201.130@tcp for MGC192.168.201.130@tcp not replayable, auto-deactivating 00000100:00080000:2.0:1788201350.048771:0:2362:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:2.0:1788201350.048782:0:2362:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff902352688000 x1875064442707328/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201351 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201350.048966:0:2362:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff902352688e00 x1875064442707840/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201356 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201350.048976:0:2362:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff90235268b480 x1875064442708224/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201361 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201350.048986:0:2362:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff902352688a80 x1875064442708608/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201366 ref 1 fl Rpc:RNQU/200/0 rc 0/-107 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:2.0:1788201350.048995:0:2362:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1788201350.048998:0:2362:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1788201350.048999:0:2362:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:2.0:1788201350.049003:0:2362:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201350.049009:0:2362:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 581 00000100:00080000:2.0:1788201350.049026:0:2362:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788201350.049071:0:2362:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234518d500 x1875064442709120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201350.049104:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442708608:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201350.049140:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442708992:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201350.049154:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442708864:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201350.049162:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442708224:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201350.051114:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442709120:192.168.201.130@tcp:250:ptlrpcd_01_01.0 00000100:00100000:3.0:1788201350.052788:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442707328:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201350.052816:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442707840:192.168.201.130@tcp:400:kworker.0 00000100:00080000:0.0:1788201350.065691:0:2358:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.201.130@tcp: connect to target with instance 0 10000000:01000000:0.0:1788201350.065708:0:2358:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:0.0:1788201350.065716:0:2358:0:(import.c:1284:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x9383f985bc4d4485 to 0x9383f985bc4d4c03 00000100:00080000:0.0:1788201350.071489:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:0.0:1788201350.071509:0:2358:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.201.130@tcp; invalidating 00000100:00100000:1.0:1788201350.071871:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442709120:192.168.201.130@tcp:250:ptlrpcd_01_01.0 00000100:00080000:0.0:1788201350.072115:0:16640:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.201.130@tcp to MGS@192.168.201.130@tcp 00000100:00100000:0.0:1788201350.072119:0:16640:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:0.0:1788201350.072123:0:16640:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:0.0:1788201350.072139:0:16640:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff902345802a00/0x4605c7c97e1c4fc9 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0x9383f985bc4d4493 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201350.072150:0:16640:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff902345802a00/0x4605c7c97e1c4fc9 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0x9383f985bc4d4493 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201350.072157:0:16640:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201350.072179:0:16640:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201350.072189:0:16640:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff902345802a00/0x4605c7c97e1c4fc9 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0x9383f985bc4d4493 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201350.072205:0:16640:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff902342db1200/0x4605c7c97e1c4fd7 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x9383f985bc4d44a1 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201350.072211:0:16640:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff902342db1200/0x4605c7c97e1c4fd7 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x9383f985bc4d44a1 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201350.072215:0:16640:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201350.072218:0:16640:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201350.072221:0:16640:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff902342db1200/0x4605c7c97e1c4fd7 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x9383f985bc4d44a1 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201350.072233:0:16640:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff902345802c00/0x4605c7c97e1c4fc2 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x9383f985bc4d448c expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201350.072240:0:16640:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff902345802c00/0x4605c7c97e1c4fc2 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x9383f985bc4d448c expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201350.072244:0:16640:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201350.072247:0:16640:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201350.072250:0:16640:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff902345802c00/0x4605c7c97e1c4fc2 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x9383f985bc4d448c expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201350.072260:0:16640:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff902351685000/0x4605c7c97e1c4fd0 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x9383f985bc4d449a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201350.072266:0:16640:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff902351685000/0x4605c7c97e1c4fd0 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x9383f985bc4d449a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201350.072272:0:16640:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:0.0:1788201350.072275:0:16640:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201350.072279:0:16640:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff902351685000/0x4605c7c97e1c4fd0 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x9383f985bc4d449a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1788201350.072294:0:16640:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:0.0:1788201350.072298:0:16640:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from RECOVER to FULL 10000000:01000000:0.0:1788201350.072300:0:16640:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:0.0:1788201350.072303:0:16640:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:0.0:1788201350.072307:0:16640:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.201.130@tcp: Connection restored to 192.168.201.130@tcp (at 192.168.201.130@tcp) 00000100:00100000:1.0:1788201350.081136:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:1.0:1788201350.790766:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201351.041051:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:0.0:1788201351.041137:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:0.0:1788201351.044352:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:0.0:1788201351.044397:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201351.046653:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788201351.052099:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788201351.105133:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 10000000:01000000:3.0:1788201351.489019:0:5601:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:3.0:1788201351.489029:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-cliir-ffff902350a11000 from 12 10000000:01000000:3.0:1788201351.489034:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788201351.489098:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff902342db0a00/0x4605c7c97e1c522a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201351.489111:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff902342db0a00/0x4605c7c97e1c522a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201351.489125:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff902342db0a00/0x4605c7c97e1c522a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201351.489159:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442709248:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788201351.489261:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe06c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201351.499197:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442709248:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201351.499233:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff902342db0a00/0x4605c7c97e1c522a lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d4c0a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201351.499249:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff902342db0a00/0x4605c7c97e1c522a lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d4c0a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201351.499341:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9f80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442709376:192.168.201.130@tcp:256:ll_cfg_requeue.0 00000100:00100000:1.0:1788201351.499438:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201351.514173:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9f80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442709376:192.168.201.130@tcp:256:ll_cfg_requeue.0 10000000:01000000:1.0:1788201351.514221:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'lustre-cliir' failed (-2). 00010000:00010000:1.0:1788201351.514232:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff902342db0a00/0x4605c7c97e1c522a lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d4c0a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201351.514244:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff902342db0a00/0x4605c7c97e1c522a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d4c0a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00000400:1.0:1788201351.514251:0:5601:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.130@tcp: IR log lustre-cliir failed, not fatal: rc = -2 10000000:01000000:1.0:1788201351.527868:0:5601:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:1.0:1788201351.527908:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-client-ffff902350a11000 from 42 10000000:01000000:1.0:1788201351.527914:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:1.0:1788201351.527985:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff902345803600/0x4605c7c97e1c5231 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201351.527998:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff902345803600/0x4605c7c97e1c5231 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201351.528010:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff902345803600/0x4605c7c97e1c5231 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201351.528040:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9f80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442709504:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201351.530367:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201351.541601:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9f80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442709504:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201351.541636:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff902345803600/0x4605c7c97e1c5231 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d4c11 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201351.541650:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff902345803600/0x4605c7c97e1c5231 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d4c11 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201351.541737:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9f80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442709632:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201351.541814:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201351.618347:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9f80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442709632:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201351.618419:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9f80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442709760:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788201351.618529:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201351.630840:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9f80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442709760:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000040:00080000:1.0:1788201351.631830:0:16641:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 42 00000020:01000000:3.0:1788201351.633362:0:5601:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 42-41 (rc=0) 10000000:01000000:3.0:1788201351.633393:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:3.0:1788201351.633406:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff902345803600/0x4605c7c97e1c5231 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d4c11 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201351.633420:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff902345803600/0x4605c7c97e1c5231 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d4c11 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201351.633437:0:5601:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:3.0:1788201351.633440:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log params-ffff902350a11000 from 7 10000000:01000000:3.0:1788201351.633442:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:3.0:1788201351.633487:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff902342db0c00/0x4605c7c97e1c5238 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201351.633495:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff902342db0c00/0x4605c7c97e1c5238 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201351.633506:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff902342db0c00/0x4605c7c97e1c5238 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201351.633533:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442709888:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788201351.633629:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0a80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201351.635774:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442709888:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788201351.635800:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff902342db0c00/0x4605c7c97e1c5238 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d4c18 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201351.635865:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff902342db0c00/0x4605c7c97e1c5238 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d4c18 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201351.635980:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442710016:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788201351.636163:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0a80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201351.644586:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442710016:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788201351.644661:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442710144:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000100:00100000:3.0:1788201351.644775:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0a80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201351.649791:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442710144:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000040:00080000:1.0:1788201351.653800:0:16642:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000020:01000000:3.0:1788201351.654371:0:5601:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:3.0:1788201351.654389:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:3.0:1788201351.654401:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff902342db0c00/0x4605c7c97e1c5238 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d4c18 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201351.654413:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff902342db0c00/0x4605c7c97e1c5238 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d4c18 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201351.654422:0:5601:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:3.0:1788201351.654425:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-sptlrpc-ffff902350bf19c0 from 1 10000000:01000000:3.0:1788201351.654427:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:3.0:1788201351.654647:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff902342db1a00/0x4605c7c97e1c523f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201351.654656:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff902342db1a00/0x4605c7c97e1c523f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201351.654667:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff902342db1a00/0x4605c7c97e1c523f lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201351.654693:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442710272:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788201351.654809:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0a80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201351.658497:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442710272:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788201351.658529:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff902342db1a00/0x4605c7c97e1c523f lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d4c1f expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201351.658547:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff902342db1a00/0x4605c7c97e1c523f lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d4c1f expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201351.658624:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442710400:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788201351.658734:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0a80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201351.662198:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442710400:192.168.201.130@tcp:501:ll_cfg_requeue.0 10000000:01000000:2.0:1788201351.662324:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:2.0:1788201351.662336:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff902342db1a00/0x4605c7c97e1c523f lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d4c1f expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201351.662347:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff902342db1a00/0x4605c7c97e1c523f lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d4c1f expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201351.809225:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201352.065038:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788201352.066209:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:0.0:1788201352.066356:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788201352.066372:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201352.066438:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:1.0:1788201352.066598:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:1.0:1788201352.129293:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:2.0:1788201352.833221:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201353.089162:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:1.0:1788201353.089306:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:2.0:1788201353.089381:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788201353.089544:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788201353.089607:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:0.0:1788201353.089728:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:1.0:1788201353.153015:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:0.0:1788201353.857127:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201354.113552:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:2.0:1788201354.113677:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788201354.113841:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788201354.117612:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:2.0:1788201354.117737:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201354.117945:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:1.0:1788201354.177715:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:1.0:1788201354.881316:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201355.137201:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:1.0:1788201355.137350:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:1.0:1788201355.137367:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201355.137474:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788201355.137666:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442710528:192.168.201.130@tcp:400:kworker.0 00000100:00080000:2.0:1788201355.137806:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201355.137813:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788201355.137861:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:1.0:1788201355.138202:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788201355.138451:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201355.138461:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442710656:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201355.139114:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201355.139127:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442710784:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201355.146465:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442710656:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201355.150484:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442710784:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201355.150994:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442710528:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201355.201215:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:0.0:1788201355.905194:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201356.161169:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788201356.163659:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:1.0:1788201356.163659:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:2.0:1788201356.163756:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:3.0:1788201356.163840:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000080:00200000:3.0:1788201356.163965:0:11804:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x403:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201356.163980:0:11804:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201356.164040:0:11804:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x403:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201356.164062:0:11804:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x404:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201356.164067:0:11804:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201356.164084:0:11804:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x404:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201356.164098:0:11804:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x405:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201356.164102:0:11804:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201356.164119:0:11804:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x405:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201356.164134:0:11804:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x406:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201356.164138:0:11804:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201356.164154:0:11804:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x406:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201356.164170:0:11804:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x407:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201356.164173:0:11804:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201356.164187:0:11804:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x407:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201356.164201:0:11804:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x408:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201356.164206:0:11804:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201356.164221:0:11804:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x408:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201356.164236:0:11804:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x409:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201356.164240:0:11804:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201356.164255:0:11804:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x409:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:0.0:1788201356.172058:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:1.0:1788201356.225221:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:3.0:1788201356.673141:0:16420:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5246 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201356.673158:0:16420:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5246 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1788201356.673266:0:2361:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff902352689180 x1875064442710912/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 664/4320 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1788201356.673267:0:16422:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c524d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201356.673287:0:16422:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c524d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201356.673337:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:1.0:1788201356.673351:0:2359:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9023458b9f80 x1875064442711040/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 664/4320 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1788201356.673831:0:16419:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5254 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201356.673869:0:16419:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5254 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201356.673928:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:0.0:1788201356.673941:0:2360:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9023488b3100 x1875064442711168/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 664/4320 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1788201356.674652:0:16427:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106600/0x4605c7c97e1c525b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 6 type: IBT flags: 0x10000000000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201356.674664:0:16427:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106600/0x4605c7c97e1c525b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201356.674782:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:2.0:1788201356.674794:0:2361:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff90234518ea00 x1875064442711296/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 664/4320 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1788201356.930796:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201357.185104:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788201357.185105:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:1.0:1788201357.185227:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:1.0:1788201357.185241:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788201357.185257:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201357.206280:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:1.0:1788201357.249075:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:0.0:1788201357.953690:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201358.209252:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:2.0:1788201358.210678:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:1.0:1788201358.211365:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788201358.211403:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788201358.211433:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201358.211470:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:2.0:1788201358.274492:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:1.0:1788201358.977097:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201359.233254:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788201359.233269:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788201359.233730:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:0.0:1788201359.233769:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:1.0:1788201359.234005:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:2.0:1788201359.234108:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:2.0:1788201359.296997:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00000100:00100000:2.0:1788201359.655024:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201359.655212:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458251648 00000100:00100000:3.0:1788201359.655229:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458251648 00000100:00100000:3.0:1788201359.655327:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1332 00000100:00100000:3.0:1788201359.655340:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518c700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:7ad49433-54fa-4e21-b756-859420d95603+3:16714:x1875064458251648:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201359.655370:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.130@tcp lock: ffff902342db0a00/0x4605c7c97e1c522a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x9383f985bc4d4c0a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201359.655483:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201359.655489:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518c700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:7ad49433-54fa-4e21-b756-859420d95603+3:16714:x1875064458251648:12345-192.168.201.130@tcp:104: Request processed in 152us (483us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201359.655497:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1332 00010000:00010000:0.0:1788201359.655993:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201359.656010:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.130@tcp lock: ffff902342db0a00/0x4605c7c97e1c522a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x9383f985bc4d4c0a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201359.656037:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db0a00 already unused, calling callback (ffffffffc12bbc20) 10000000:00010000:0.0:1788201359.656043:0:5599:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.130@tcp lock: ffff902342db0a00/0x4605c7c97e1c522a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x9383f985bc4d4c0a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201359.656062:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff902342db0a00/0x4605c7c97e1c522a lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x9383f985bc4d4c0a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201359.656070:0:5599:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff902342db0a00/0x4605c7c97e1c522a lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x9383f985bc4d4c0a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201359.656076:0:5599:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201359.656145:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.130@tcp lock: ffff902342db0a00/0x4605c7c97e1c522a lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x9383f985bc4d4c0a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201359.656152:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201359.656178:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.130@tcp lock: ffff902342db0a00/0x4605c7c97e1c522a lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x9383f985bc4d4c0a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201359.656186:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff902342db0a00/0x4605c7c97e1c522a lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x9383f985bc4d4c0a expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201359.656208:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201359.656210:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201359.656212:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201359.656238:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442711424:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201359.668636:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442711424:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201360.001444:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201360.257073:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788201360.257125:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00080000:0.0:1788201360.257399:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201360.257406:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788201360.257506:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:0.0:1788201360.257532:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442711552:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201360.257676:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201360.257685:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442711680:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201360.259705:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:0.0:1788201360.261634:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788201360.262067:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00010000:00010000:0.0:1788201360.266943:0:2359:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00100000:0.0:1788201360.266975:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442711680:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201360.267007:0:2359:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 2 blwis in pool 00000100:00100000:0.0:1788201360.267013:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442711552:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201360.267052:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1788201360.267074:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201360.267081:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201360.267083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201360.267084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201360.267334:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201360.321049:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 10000000:01000000:1.0:1788201360.833054:0:5601:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788201360.833068:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-cliir-ffff902350a11000 from 12 10000000:01000000:1.0:1788201360.833076:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788201360.833158:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff902345802000/0x4605c7c97e1c5262 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201360.833175:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff902345802000/0x4605c7c97e1c5262 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201360.833189:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff902345802000/0x4605c7c97e1c5262 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201360.833227:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458bb800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442711808:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201360.833396:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fab40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201360.836821:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458bb800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442711808:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201360.837006:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff902345802000/0x4605c7c97e1c5262 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d4d22 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201360.837026:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff902345802000/0x4605c7c97e1c5262 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d4d22 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201360.837139:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458bb800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442711936:192.168.201.130@tcp:256:ll_cfg_requeue.0 00000100:00100000:1.0:1788201360.837265:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fab40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201360.860056:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458bb800 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442711936:192.168.201.130@tcp:256:ll_cfg_requeue.0 00000100:00080000:1.0:1788201360.860142:0:5601:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 9, was 0): CONNECTING 00010000:00080000:1.0:1788201360.860161:0:5601:0:(ldlm_lib.c:85:import_set_conn()) imp ffff902348897000@lustre-MDT0000-mdc-ffff902350a11000: found existing conn 192.168.201.130@tcp, moved to head 10000000:01000000:1.0:1788201360.860208:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:1.0:1788201360.860221:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff902345802000/0x4605c7c97e1c5262 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d4d22 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201360.860233:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff902345802000/0x4605c7c97e1c5262 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d4d22 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201361.031578:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201361.281162:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:1.0:1788201361.281248:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788201361.281444:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788201361.281617:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:0.0:1788201361.282452:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:1.0:1788201361.301754:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201361.349099:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:2.0:1788201362.049115:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201362.305079:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:0.0:1788201362.305281:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788201362.305299:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:1.0:1788201362.305510:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:2.0:1788201362.305707:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:0.0:1788201362.305771:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788201362.369032:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:0.0:1788201363.073190:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201363.329001:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:2.0:1788201363.329002:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:0.0:1788201363.329361:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788201363.329403:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:2.0:1788201363.329639:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788201363.329781:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201363.393041:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:3.0:1788201364.097199:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201364.353140:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:0.0:1788201364.353399:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201364.353448:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:2.0:1788201364.353532:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788201364.353579:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788201364.353693:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:3.0:1788201364.417012:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:0.0:1788201365.121367:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201365.377068:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788201365.377234:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:0.0:1788201365.377735:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788201365.377899:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201365.377905:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1788201365.377979:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:0.0:1788201365.378101:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442712064:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201365.378209:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442712320:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201365.378244:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201365.378251:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442712192:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201365.378423:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:1.0:1788201365.379157:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:0.0:1788201365.406209:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442712064:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201365.406235:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442712320:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201365.413461:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442712192:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201365.441398:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:2.0:1788201366.146406:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201366.401018:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:0.0:1788201366.405590:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:0.0:1788201366.405621:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:0.0:1788201366.405633:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788201366.405644:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788201366.409986:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:3.0:1788201366.464993:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:0.0:1788201367.169147:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201367.425137:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:3.0:1788201367.425251:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:0.0:1788201367.425389:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201367.425440:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:2.0:1788201367.425483:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788201367.425543:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788201367.489016:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:3.0:1788201368.193225:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201368.449127:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788201368.449604:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:2.0:1788201368.449821:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:0.0:1788201368.449868:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:0.0:1788201368.451313:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788201368.451374:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:3.0:1788201368.513294:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00010000:00010000:3.0:1788201369.218062:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201369.476019:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201369.476081:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788201369.476253:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:3.0:1788201369.490725:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:1.0:1788201369.491473:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788201369.492833:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:2.0:1788201369.540709:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00000080:00200008:1.0:1788201369.922866:0:16741:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201369.927103:0:16741:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201369.927131:0:16741:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201369.927151:0:16741:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:0.0:1788201369.927167:0:16741:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:0.0:1788201369.927175:0:16741:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201369.927184:0:16741:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201369.927187:0:16741:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:0.0:1788201369.927194:0:16741:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181077cb8 00010000:00010000:0.0:1788201369.927198:0:16741:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:0.0:1788201369.927206:0:16741:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200008:0.0:1788201369.927840:0:16741:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788201369.927847:0:16741:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201369.927856:0:16741:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201369.927864:0:16741:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:0.0:1788201369.927910:0:16741:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:0.0:1788201369.927918:0:16741:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201369.927923:0:16741:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201369.927925:0:16741:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:0.0:1788201369.927929:0:16741:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181077cb8 00010000:00010000:0.0:1788201369.927932:0:16741:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:0.0:1788201369.927939:0:16741:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:0.0:1788201369.928044:0:16741:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201369.928053:0:16741:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201369.928060:0:16741:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:0.0:1788201369.928070:0:16741:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:0.0:1788201369.928076:0:16741:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201369.928080:0:16741:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201369.928083:0:16741:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:0.0:1788201369.928086:0:16741:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181077bb0 00010000:00010000:0.0:1788201369.928089:0:16741:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:0.0:1788201369.928097:0:16741:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:0.0:1788201369.928106:0:16741:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 44 00000080:00200008:0.0:1788201369.928112:0:16741:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00000080:00200080:0.0:1788201369.928207:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201369.928222:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00800000:00000004:0.0:1788201369.928229:0:16741:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: age 592, max_age 645 00000100:00080000:0.0:1788201369.928274:0:16741:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9023488b1880 x1875064442712448/t0(0) o41->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201369.928284:0:16741:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352616600 going to sleep for 0 seconds 00010000:00010000:2.0:1788201370.241304:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201370.497076:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 0 seconds 00000100:00100000:0.0:1788201370.497230:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788201370.497253:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201350/real 1788201350] req@ffff90235268b800 x1875064442708736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201370 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201370.497284:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442708736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201370 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201370.497302:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201370.497312:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201370.497316:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442708736:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201370.497396:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201370.497445:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 0 seconds 00000100:00080000:3.0:1788201370.497638:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201370.497643:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201370.497645:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201370.497650:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 0 00000100:00080000:3.0:1788201370.497663:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201370.497673:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201370.497689:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442712704/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201370.497816:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442712576:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201370.497939:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201370.497945:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442712832:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201370.498328:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201370.498338:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442712960:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201370.498787:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442712704:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201370.499516:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 0 seconds 00000100:00100000:2.0:1788201370.499780:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788201370.504828:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442712960:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201370.505499:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442712832:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201370.514319:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442712576:192.168.201.130@tcp:400:kworker.0 00010000:00080000:0.0:1788201370.517477:0:2358:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff902352689c00 x1875064442712704/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/416 e 0 to 0 dl 1788201380 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201370.517503:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 12884901940 gen 2 00000100:00080000:0.0:1788201370.517581:0:2358:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff902350a11000: connect to target with instance 4 00000100:00080000:0.0:1788201370.517613:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:0.0:1788201370.517619:0:2358:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788201370.517621:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 12884901940 gen 2 00000100:00080000:0.0:1788201370.517624:0:2358:0:(recover.c:53:ptlrpc_replay_next()) import ffff902348897000 from lustre-MDT0000_UUID committed 12884901940 last 0 00000100:00080000:0.0:1788201370.517626:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:2.0:1788201370.518365:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442712704:192.168.201.130@tcp:38:kworker.0 00010000:00080000:0.0:1788201370.523340:0:16756:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff902350a11000 to lustre-MDT0000_UUID@192.168.201.130@tcp 00010000:00010000:0.0:1788201370.523349:0:16756:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff902350a11000 (17) 00000080:00200000:0.0:1788201370.524014:0:16756:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x406:0x0]: losing layout lock 00010000:00010000:0.0:1788201370.524061:0:16756:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c51cf lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d4b00 expref: -99 pid: 16422 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201370.524096:0:16756:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x409:0x0]: losing layout lock 00010000:00010000:0.0:1788201370.524111:0:16756:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c5215 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d4bc4 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201370.524131:0:16756:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x408:0x0]: losing layout lock 00010000:00010000:0.0:1788201370.524143:0:16756:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c520e lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d4ba8 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201370.524160:0:16756:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x407:0x0]: losing layout lock 00010000:00010000:0.0:1788201370.524172:0:16756:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c5207 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d4b8c expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201370.524187:0:16756:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x406:0x0]: losing layout lock 00010000:00010000:0.0:1788201370.524196:0:16756:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5200 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d4b70 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201370.524211:0:16756:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x405:0x0]: losing layout lock 00010000:00010000:0.0:1788201370.524222:0:16756:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c51f9 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d4b54 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201370.524241:0:16756:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x404:0x0]: losing layout lock 00010000:00010000:0.0:1788201370.524254:0:16756:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c51f2 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d4b38 expref: -99 pid: 16427 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201370.524270:0:16756:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x406:0x0]: losing layout lock 00010000:00010000:0.0:1788201370.524281:0:16756:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5151 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d4aba expref: -99 pid: 16421 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201370.524300:0:16756:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x403:0x0]: losing layout lock 00010000:00010000:0.0:1788201370.524311:0:16756:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c509b lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d482f expref: -99 pid: 16393 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201370.524333:0:16756:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x408:0x0]: losing layout lock 00010000:00010000:0.0:1788201370.524342:0:16756:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c50e1 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d4924 expref: -99 pid: 16398 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201370.524361:0:16756:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x407:0x0]: losing layout lock 00010000:00010000:0.0:1788201370.524371:0:16756:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d400/0x4605c7c97e1c50d3 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d48f3 expref: -99 pid: 16397 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201370.524388:0:16756:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x409:0x0]: losing layout lock 00010000:00010000:0.0:1788201370.524398:0:16756:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c50ef lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d4955 expref: -99 pid: 16399 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201370.525276:0:16756:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x405:0x0]: losing layout lock 00010000:00010000:0.0:1788201370.525310:0:16756:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c50b7 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d4891 expref: -99 pid: 16395 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201370.525343:0:16756:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x406:0x0]: losing layout lock 00010000:00010000:0.0:1788201370.525356:0:16756:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c50c5 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d48c2 expref: -99 pid: 16396 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201370.525379:0:16756:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x404:0x0]: losing layout lock 00010000:00010000:0.0:1788201370.525390:0:16756:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c50a9 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x9383f985bc4d4860 expref: -99 pid: 16394 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201370.525407:0:16756:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c508d lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x11/0x0 rrc: 6 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x9383f985bc4d47fe expref: -99 pid: 16393 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201370.526195:0:16756:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff902350a11000 00010000:00010000:0.0:1788201370.527108:0:16756:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d47a3 expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000100:00100000:2.0:1788201370.527206:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442713088:192.168.201.130@tcp:101:ldlm_lock_repla.0 00010000:00080000:0.0:1788201370.527319:0:16756:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:2.0:1788201370.530837:0:2358:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000100:00080000:2.0:1788201370.530855:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:2.0:1788201370.531266:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442713088:192.168.201.130@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788201370.531296:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442713216:192.168.201.130@tcp:400:ptlrpcd_rcv.0 00000100:00100000:2.0:1788201370.561257:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 0 seconds 00000100:00080000:2.0:1788201370.601551:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:2.0:1788201370.601571:0:2358:0:(client.c:3234:ptlrpc_resend_req()) @@@ going to resend req@ffff90234518fb80 x1875064442713216/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201386 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201370.601581:0:2358:0:(client.c:3243:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff90234518fb80 x1875064442713216/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201386 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201370.601594:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:0.0:1788201370.602522:0:2358:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff902350a11000: Connection restored to 192.168.201.130@tcp (at 192.168.201.130@tcp) 00000100:00080000:0.0:1788201370.622969:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000002a5d3e6): req@ffff9023488b3100 x1875064442711168/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 664/4320 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201370.622999:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000041905aa6): req@ffff9023458b9f80 x1875064442711040/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 664/4320 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201370.623011:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000085c70249): req@ffff9023488b1880 x1875064442712448/t0(0) o41->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201370.623375:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000038cbeb72): req@ffff9023488b3480 x1875064442706432/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201342 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201370.623399:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000057998f7f): req@ffff9023458c4e00 x1875064442706176/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201342 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201370.623410:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000fe1e8411): req@ffff902352689500 x1875064442704256/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201342 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201370.623423:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000b9f77c6d): req@ffff9023458c5880 x1875064442705920/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201342 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201370.623436:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000006149b2e6): req@ffff9023458c6d80 x1875064442705536/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201342 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201370.623449:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000004b5594fa): req@ffff9023488b1180 x1875064442704000/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201342 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201370.623461:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000309c5095): req@ffff90234518ea00 x1875064442711296/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 664/4320 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201370.623473:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000309c5095): req@ffff902352689180 x1875064442710912/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 664/4320 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201370.623482:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000c307f2e9): req@ffff9023458b8380 x1875064442705152/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 1 dl 1788201342 ref 2 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'chmod.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201370.623496:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442713216:192.168.201.130@tcp:400:ptlrpcd_rcv.0 00000100:00100000:2.0:1788201370.625176:0:16741:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352616600 going to sleep for 11 seconds 00000100:00080000:2.0:1788201370.625203:0:16414:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442704256 new x1875064442713344 00000100:00100000:2.0:1788201370.625230:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 11 seconds 00000100:00080000:2.0:1788201370.625248:0:16412:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442704000 new x1875064442713472 00000100:00100000:2.0:1788201370.625269:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584483c0 going to sleep for 11 seconds 00000100:00080000:2.0:1788201370.625287:0:16413:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442705152 new x1875064442713600 00000100:00100000:2.0:1788201370.625305:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 11 seconds 00000100:00080000:2.0:1788201370.625339:0:16415:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442705920 new x1875064442713728 00000100:00100000:2.0:1788201370.625359:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 11 seconds 00000100:00080000:2.0:1788201370.626487:0:16416:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442706176 new x1875064442713856 00000100:00100000:2.0:1788201370.626552:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 11 seconds 00000100:00080000:2.0:1788201370.626635:0:16417:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442705536 new x1875064442713984 00000100:00100000:2.0:1788201370.626662:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57240 going to sleep for 11 seconds 00000100:00080000:3.0:1788201370.635038:0:16411:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442706432 new x1875064442714112 00000100:00100000:3.0:1788201370.635185:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352651300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201370.664259:0:16741:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16741:1875064442712448:192.168.201.130@tcp:41:lfs.0 00800000:00000004:0.0:1788201370.664289:0:16741:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: update ffff902350be5190 cache blocks 17239/17271 objects 531669/532010 00000080:00200080:0.0:1788201370.664385:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664403:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664411:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664416:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664420:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664424:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664428:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664432:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664435:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664439:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664443:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664446:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664449:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664452:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664456:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664459:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664462:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664466:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664469:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664473:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664476:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664479:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664482:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664485:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664489:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664493:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664505:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664509:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664514:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664516:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664520:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664524:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664528:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664531:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664535:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664538:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664542:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664545:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664548:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664552:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664555:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664558:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664562:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664565:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664569:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664572:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664576:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664579:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664583:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664587:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664591:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664594:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664608:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664612:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664616:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664619:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664624:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664627:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664631:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664634:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664638:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664642:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664646:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664649:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664653:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664656:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664659:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664663:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664666:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664669:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664673:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664677:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664681:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664684:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664688:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664691:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664694:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664697:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664701:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664705:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664709:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664713:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664726:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664731:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664736:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664739:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664742:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664746:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664750:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664754:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664758:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664762:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664766:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664770:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664773:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664777:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664781:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664783:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664787:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664790:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664794:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664797:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664801:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664805:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664809:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664813:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664817:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664820:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664823:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664826:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664829:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664832:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664835:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664838:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664842:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664844:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664847:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664851:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664855:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664858:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664861:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664864:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664868:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664872:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664913:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664920:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664924:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00800000:00000080:0.0:1788201370.664927:0:16741:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.664932:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.664938:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00020000:00000004:0.0:1788201370.664946:0:16741:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: age 592, max_age 646 00000100:00100000:0.0:1788201370.664984:0:16741:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16741:1875064442714240:192.168.201.130@tcp:13:lfs.0 00000100:00100000:0.0:1788201370.665050:0:16741:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235279ef00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201370.676155:0:16741:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16741:1875064442714240:192.168.201.130@tcp:13:lfs.0 00020000:00000004:0.0:1788201370.676186:0:16741:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: update ffff902350be1e10 cache blocks 3667/3683 objects 117728/118665 00000080:00200080:0.0:1788201370.676250:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.676266:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00020000:00000004:0.0:1788201370.676275:0:16741:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: age 592, max_age 646 00000100:00100000:0.0:1788201370.676310:0:16741:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16741:1875064442714368:192.168.201.130@tcp:13:lfs.0 00000100:00100000:0.0:1788201370.677097:0:16741:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235279ef00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201370.683713:0:16741:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16741:1875064442714368:192.168.201.130@tcp:13:lfs.0 00020000:00000004:0.0:1788201370.684298:0:16741:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: update ffff902351d65190 cache blocks 3666/3683 objects 117792/118185 00000080:00200080:0.0:1788201370.684389:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684411:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684418:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684423:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684428:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684432:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684436:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684439:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684443:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684447:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684450:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684454:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684457:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684461:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684464:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684468:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684472:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684476:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684479:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684483:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684487:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684491:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684494:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684641:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684648:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684652:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684656:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684660:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684664:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684667:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684671:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684675:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684679:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684682:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684686:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684689:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684692:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684695:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684699:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684703:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684707:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684710:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684785:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684790:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684794:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684798:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684802:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684805:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684809:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684812:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684816:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684819:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684824:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684827:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684831:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684835:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684839:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684842:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684846:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684850:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684854:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684857:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684861:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684865:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684868:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684871:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684897:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684902:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684907:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684910:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684914:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684919:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684923:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684926:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684930:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684934:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684937:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684941:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684945:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684948:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684952:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684955:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684958:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684961:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684966:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684969:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684972:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684975:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684979:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684982:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684985:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684989:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684993:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.684996:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.684999:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.685003:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.685007:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.685011:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.685014:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.685017:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.685022:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.685025:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.685029:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.685032:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.685036:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.685039:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.685043:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.685046:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.685050:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.685054:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.685058:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.685061:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.685065:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.685068:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.685072:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.685075:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.685079:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.685082:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.685085:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.685087:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.685091:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.685095:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.685098:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.685100:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200080:0.0:1788201370.685104:0:16741:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcd1192310 00020000:00000080:0.0:1788201370.685107:0:16741:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262d800 uarg=0000000000000000 00000080:00200008:0.0:1788201370.685121:0:16741:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00000100:00100000:0.0:1788201370.787838:0:16412:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1180 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16412:1875064442704000:192.168.201.130@tcp:36:chmod.0 00000080:00200000:0.0:1788201370.787961:0:16412:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x404:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:0.0:1788201370.787984:0:16412:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x404:0x0](ffff9023449a7448) 00000002:00010000:0.0:1788201370.787996:0:16412:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x404:0x0]) in obj [0x200000bd1:0x404:0x0], intent: layout flags 0 00010000:00010000:0.0:1788201370.788049:0:16412:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5269 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 16412 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201370.788059:0:16412:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5269 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16412 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201370.788069:0:16412:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5269 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16412 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201370.788088:0:16412:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1180 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16412:1875064442714496:192.168.201.130@tcp:101:chmod.0 00000100:00100000:3.0:1788201370.795035:0:16416:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c4e00 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16416:1875064442706176:192.168.201.130@tcp:36:chmod.0 00000080:00200000:3.0:1788201370.795140:0:16416:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x408:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:3.0:1788201370.795163:0:16416:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x408:0x0](ffff9023449ab248) 00000002:00010000:3.0:1788201370.795175:0:16416:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x408:0x0]) in obj [0x200000bd1:0x408:0x0], intent: layout flags 0 00010000:00010000:3.0:1788201370.795225:0:16416:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1000/0x4605c7c97e1c5270 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 16416 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.795236:0:16416:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1000/0x4605c7c97e1c5270 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16416 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.795247:0:16416:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1000/0x4605c7c97e1c5270 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16416 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201370.795268:0:16416:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689f80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16416:1875064442714624:192.168.201.130@tcp:101:chmod.0 00000100:00100000:3.0:1788201370.795338:0:16416:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe06c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201370.795366:0:16417:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c6d80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16417:1875064442705536:192.168.201.130@tcp:36:chmod.0 00000080:00200000:3.0:1788201370.795404:0:16417:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x409:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:3.0:1788201370.795414:0:16417:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x409:0x0](ffff9023449adb88) 00000002:00010000:3.0:1788201370.795419:0:16417:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x409:0x0]) in obj [0x200000bd1:0x409:0x0], intent: layout flags 0 00010000:00010000:3.0:1788201370.795440:0:16417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c5277 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 16417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.795449:0:16417:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c5277 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16417 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.795456:0:16417:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c5277 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16417 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201370.795467:0:16417:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16417:1875064442714752:192.168.201.130@tcp:101:chmod.0 00000100:00100000:3.0:1788201370.795490:0:16417:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201370.795517:0:16411:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b3480 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16411:1875064442706432:192.168.201.130@tcp:36:chmod.0 00000080:00200000:3.0:1788201370.795550:0:16411:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x403:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:3.0:1788201370.795558:0:16411:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x403:0x0](ffff90234497ec08) 00000002:00010000:3.0:1788201370.795563:0:16411:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x403:0x0]) in obj [0x200000bd1:0x403:0x0], intent: layout flags 0 00010000:00010000:3.0:1788201370.795581:0:16411:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c527e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 16411 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.795587:0:16411:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c527e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16411 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.795606:0:16411:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c527e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16411 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201370.795622:0:16411:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16411:1875064442714880:192.168.201.130@tcp:101:chmod.0 00000100:00100000:3.0:1788201370.795645:0:16411:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0840 going to sleep for 11 seconds 00000100:00100000:3.0:1788201370.795669:0:16414:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16414:1875064442704256:192.168.201.130@tcp:36:chmod.0 00010000:00010000:1.0:1788201370.796725:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c524d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796740:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c524d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796746:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c524d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796753:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c524d lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796761:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c524d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796769:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345803400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201370.796773:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c524d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796781:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c524d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796788:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c524d lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x20e400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796794:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201370.796801:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c524d lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796806:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c524d lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 16422 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201370.796835:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:0:1875064442711040:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:1.0:1788201370.796891:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5254 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796903:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5254 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796910:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5254 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796915:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5254 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796921:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5254 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796925:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902343730e00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201370.796928:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5254 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796933:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5254 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796938:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5254 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x20e400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796943:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201370.796946:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5254 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.796950:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c5254 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 16419 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201370.796966:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:0:1875064442711168:192.168.201.130@tcp:63:ll_sa.0 00000080:00200000:3.0:1788201370.800419:0:16414:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x406:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:3.0:1788201370.800452:0:16414:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x406:0x0](ffff902344979148) 00000002:00010000:3.0:1788201370.800465:0:16414:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x406:0x0]) in obj [0x200000bd1:0x406:0x0], intent: layout flags 0 00000100:00100000:2.0:1788201370.800690:0:16415:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458c5880 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16415:1875064442705920:192.168.201.130@tcp:36:chmod.0 00000080:00200000:2.0:1788201370.800786:0:16415:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x407:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:2.0:1788201370.800795:0:16415:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x407:0x0](ffff9023449b4b08) 00000002:00010000:2.0:1788201370.800800:0:16415:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x407:0x0]) in obj [0x200000bd1:0x407:0x0], intent: layout flags 0 00010000:00010000:2.0:1788201370.800851:0:16415:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106e00/0x4605c7c97e1c5285 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 16415 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201370.800862:0:16415:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106e00/0x4605c7c97e1c5285 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16415 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201370.800874:0:16415:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106e00/0x4605c7c97e1c5285 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16415 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201370.800916:0:16415:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518f480 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16415:1875064442715008:192.168.201.130@tcp:101:chmod.0 00000100:00100000:2.0:1788201370.800999:0:16415:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342e36600 going to sleep for 11 seconds 00010000:00010000:3.0:1788201370.802252:0:16414:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c528c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 16414 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.802266:0:16414:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c528c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16414 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.802277:0:16414:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c528c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16414 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201370.802298:0:16414:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16414:1875064442715136:192.168.201.130@tcp:101:chmod.0 00000100:00100000:3.0:1788201370.802350:0:16414:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201370.802402:0:16413:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b8380 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16413:1875064442705152:192.168.201.130@tcp:36:chmod.0 00000080:00200000:3.0:1788201370.802472:0:16413:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x405:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:3.0:1788201370.804872:0:16413:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x405:0x0](ffff9023449b5348) 00000002:00010000:3.0:1788201370.804907:0:16413:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x405:0x0]) in obj [0x200000bd1:0x405:0x0], intent: layout flags 0 00010000:00010000:3.0:1788201370.804959:0:16413:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5293 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 16413 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.804971:0:16413:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5293 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16413 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.804986:0:16413:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5293 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16413 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201370.805010:0:16413:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16413:1875064442715264:192.168.201.130@tcp:101:chmod.0 00000100:00100000:3.0:1788201370.805078:0:16413:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0540 going to sleep for 11 seconds 00000100:00100000:3.0:1788201370.810084:0:16417:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16417:1875064442714752:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201370.810120:0:16417:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c5277 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4ed4 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.810136:0:16417:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c5277 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ed4 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201370.810152:0:16417:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff902352688000 x1875064442714752/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201386 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201370.810163:0:16417:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c5277 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ed4 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201370.810177:0:16417:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c5277 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ed4 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.810189:0:16417:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1033 (0 0) 00000080:00200000:3.0:1788201370.810196:0:16417:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x409:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1788201370.810216:0:16417:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x409:0x0](ffff9023449adb88) for lock 0x4605c7c97e1c5277 00000080:00010000:3.0:1788201370.810225:0:16417:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5277 from it ffffa0b181c33ad8 00010000:00010000:3.0:1788201370.810229:0:16417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c5277 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4ed4 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.810246:0:16417:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c5277 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ed4 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201370.810269:0:16417:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c5277 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ed4 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201370.810281:0:16417:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c5277 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ed4 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201370.810290:0:16417:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x409:0x0](0000000002670e8b) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c5277 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ed4 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201370.810354:0:16417:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x409:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:3.0:1788201370.810375:0:16417:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c5277 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4ed4 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201370.810383:0:16417:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c5277 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4ed4 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:3.0:1788201370.810403:0:16417:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201370.810495:0:16411:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16411:1875064442714880:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201370.810507:0:16411:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c527e lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4edb expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.810516:0:16411:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c527e lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4edb expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201370.810532:0:16411:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff902352688e00 x1875064442714880/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201386 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201370.810542:0:16411:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c527e lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4edb expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201370.810552:0:16411:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c527e lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4edb expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.810560:0:16411:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1027 (0 0) 00000080:00200000:3.0:1788201370.810565:0:16411:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x403:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1788201370.810575:0:16411:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x403:0x0](ffff90234497ec08) for lock 0x4605c7c97e1c527e 00000080:00010000:3.0:1788201370.810580:0:16411:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c527e from it ffffa0b181c03ad8 00010000:00010000:3.0:1788201370.810583:0:16411:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c527e lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4edb expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.810590:0:16411:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c527e lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4edb expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201370.810685:0:16416:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689f80 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16416:1875064442714624:192.168.201.130@tcp:101:chmod.0 00010000:00010000:1.0:1788201370.810708:0:16416:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1000/0x4605c7c97e1c5270 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4ecd expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.810728:0:16416:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1000/0x4605c7c97e1c5270 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ecd expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201370.810738:0:16416:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff902352689f80 x1875064442714624/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201386 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201370.810747:0:16416:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1000/0x4605c7c97e1c5270 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ecd expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201370.810756:0:16416:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1000/0x4605c7c97e1c5270 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ecd expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.810764:0:16416:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1032 (0 0) 00000080:00200000:1.0:1788201370.810769:0:16416:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x408:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1788201370.810778:0:16416:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x408:0x0](ffff9023449ab248) for lock 0x4605c7c97e1c5270 00000080:00010000:1.0:1788201370.810783:0:16416:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5270 from it ffffa0b181c2bad8 00010000:00010000:1.0:1788201370.810786:0:16416:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1000/0x4605c7c97e1c5270 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4ecd expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201370.810794:0:16416:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1000/0x4605c7c97e1c5270 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ecd expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201370.810807:0:16416:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1000/0x4605c7c97e1c5270 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ecd expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201370.810816:0:16416:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1000/0x4605c7c97e1c5270 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ecd expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201370.810823:0:16416:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x408:0x0](00000000acc1f1d8) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1000/0x4605c7c97e1c5270 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ecd expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201370.810862:0:16416:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x408:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:1.0:1788201370.810896:0:16416:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1000/0x4605c7c97e1c5270 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4ecd expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201370.810905:0:16416:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1000/0x4605c7c97e1c5270 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4ecd expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:1.0:1788201370.810918:0:16416:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:1.0:1788201370.810987:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442715520:192.168.201.130@tcp:2:chmod.0 00000100:00100000:0.0:1788201370.811286:0:16412:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235279ef00 going to sleep for 11 seconds 00010000:00010000:3.0:1788201370.813957:0:16411:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c527e lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4edb expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201370.813974:0:16411:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c527e lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4edb expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201370.813985:0:16411:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x403:0x0](000000003e1b269b) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c527e lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4edb expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201370.814071:0:16411:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x403:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:3.0:1788201370.814095:0:16411:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c527e lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4edb expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201370.814104:0:16411:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c527e lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4edb expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:3.0:1788201370.814125:0:16411:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201370.814258:0:16415:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518f480 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16415:1875064442715008:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201370.814284:0:16415:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106e00/0x4605c7c97e1c5285 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4ee2 expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.814298:0:16415:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106e00/0x4605c7c97e1c5285 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ee2 expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201370.814316:0:16415:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff90234518f480 x1875064442715008/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201386 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201370.814328:0:16415:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106e00/0x4605c7c97e1c5285 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ee2 expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201370.814339:0:16415:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106e00/0x4605c7c97e1c5285 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ee2 expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.814349:0:16415:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1031 (0 0) 00000080:00200000:3.0:1788201370.814355:0:16415:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x407:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1788201370.814369:0:16415:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x407:0x0](ffff9023449b4b08) for lock 0x4605c7c97e1c5285 00000080:00010000:3.0:1788201370.814375:0:16415:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5285 from it ffffa0b181c23ad8 00010000:00010000:3.0:1788201370.814378:0:16415:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106e00/0x4605c7c97e1c5285 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4ee2 expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.814386:0:16415:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106e00/0x4605c7c97e1c5285 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ee2 expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201370.814398:0:16415:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106e00/0x4605c7c97e1c5285 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ee2 expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201370.814406:0:16415:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106e00/0x4605c7c97e1c5285 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ee2 expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201370.814413:0:16415:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x407:0x0](00000000d67d3de0) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106e00/0x4605c7c97e1c5285 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ee2 expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201370.814446:0:16415:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x407:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:3.0:1788201370.814459:0:16415:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106e00/0x4605c7c97e1c5285 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4ee2 expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201370.814467:0:16415:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106e00/0x4605c7c97e1c5285 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4ee2 expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:3.0:1788201370.815888:0:16415:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201370.816006:0:16414:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16414:1875064442715136:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201370.816028:0:16414:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c528c lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4ee9 expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816045:0:16414:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c528c lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ee9 expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201370.816057:0:16414:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff902352689500 x1875064442715136/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201386 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201370.816067:0:16414:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c528c lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ee9 expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201370.816077:0:16414:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c528c lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ee9 expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816087:0:16414:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1030 (0 0) 00000080:00200000:3.0:1788201370.816093:0:16414:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x406:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1788201370.816108:0:16414:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x406:0x0](ffff902344979148) for lock 0x4605c7c97e1c528c 00000080:00010000:3.0:1788201370.816114:0:16414:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c528c from it ffffa0b181c1bad8 00010000:00010000:3.0:1788201370.816118:0:16414:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c528c lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4ee9 expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816127:0:16414:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c528c lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ee9 expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201370.816144:0:16414:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c528c lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ee9 expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201370.816153:0:16414:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c528c lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ee9 expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201370.816166:0:16414:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x406:0x0](000000005cda43cf) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c528c lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ee9 expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201370.816215:0:16414:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x406:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:3.0:1788201370.816232:0:16414:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c528c lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4ee9 expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201370.816240:0:16414:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c528c lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4ee9 expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:3.0:1788201370.816253:0:16414:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201370.816299:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442715392:192.168.201.130@tcp:2:chmod.0 00000100:00100000:3.0:1788201370.816371:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442715776:192.168.201.130@tcp:2:chmod.0 00000100:00100000:3.0:1788201370.816396:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201370.816402:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442715648:192.168.201.130@tcp:2:chmod.0 00000100:00100000:3.0:1788201370.816422:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442715904:192.168.201.130@tcp:2:chmod.0 00010000:00010000:3.0:1788201370.816478:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106600/0x4605c7c97e1c525b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x0 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816487:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106600/0x4605c7c97e1c525b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816492:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106600/0x4605c7c97e1c525b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816502:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106600/0x4605c7c97e1c525b lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816508:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106600/0x4605c7c97e1c525b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816515:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345106600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201370.816519:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106600/0x4605c7c97e1c525b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816526:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106600/0x4605c7c97e1c525b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816533:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106600/0x4605c7c97e1c525b lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x20e400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816541:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201370.816546:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106600/0x4605c7c97e1c525b lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816552:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106600/0x4605c7c97e1c525b lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 16427 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201370.816576:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:0:1875064442711296:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:3.0:1788201370.816592:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5246 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x0 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816611:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5246 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816620:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5246 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816626:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5246 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816632:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5246 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816637:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db0400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201370.816639:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5246 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816644:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5246 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x206400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816649:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5246 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x20e400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816653:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201370.816656:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5246 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.816661:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c5246 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 16420 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201370.816680:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:0:1875064442710912:192.168.201.130@tcp:63:ll_sa.0 00000100:00100000:3.0:1788201370.822017:0:16413:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16413:1875064442715264:192.168.201.130@tcp:101:chmod.0 00010000:00010000:3.0:1788201370.822060:0:16413:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5293 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4ef0 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.822078:0:16413:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5293 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ef0 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201370.822093:0:16413:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff90235268b480 x1875064442715264/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201386 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201370.822105:0:16413:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5293 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ef0 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201370.822121:0:16413:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5293 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ef0 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.822133:0:16413:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1029 (0 0) 00000080:00200000:3.0:1788201370.822140:0:16413:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x405:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1788201370.822159:0:16413:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x405:0x0](ffff9023449b5348) for lock 0x4605c7c97e1c5293 00000080:00010000:3.0:1788201370.822167:0:16413:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5293 from it ffffa0b181c13ad8 00010000:00010000:3.0:1788201370.822171:0:16413:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5293 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4ef0 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201370.822180:0:16413:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5293 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ef0 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201370.822203:0:16413:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5293 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ef0 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201370.822215:0:16413:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5293 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ef0 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201370.822229:0:16413:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x405:0x0](00000000484282c9) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5293 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4ef0 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201370.822291:0:16413:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x405:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:3.0:1788201370.822310:0:16413:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5293 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4ef0 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201370.822319:0:16413:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5293 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4ef0 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:3.0:1788201370.822338:0:16413:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201370.822617:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201370.822633:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442716032:192.168.201.130@tcp:2:chmod.0 00000100:00100000:2.0:1788201370.835871:0:16412:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1180 pname:cluuid:pid:xid:nid:opc:job chmod:35604eb4-baf9-45be-a20a-4883a2cd8cf9:16412:1875064442714496:192.168.201.130@tcp:101:chmod.0 00010000:00010000:2.0:1788201370.835923:0:16412:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5269 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4f28 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201370.835937:0:16412:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5269 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4f28 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201370.835952:0:16412:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9023488b1180 x1875064442714496/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201386 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'chmod.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201370.835963:0:16412:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5269 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4f28 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201370.835974:0:16412:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5269 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4f28 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201370.835987:0:16412:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1028 (0 0) 00000080:00200000:2.0:1788201370.835993:0:16412:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x404:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1788201370.836010:0:16412:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x404:0x0](ffff9023449a7448) for lock 0x4605c7c97e1c5269 00000080:00010000:2.0:1788201370.836018:0:16412:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5269 from it ffffa0b180d4bad8 00010000:00010000:2.0:1788201370.836023:0:16412:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5269 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4f28 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201370.836033:0:16412:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5269 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4f28 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201370.836058:0:16412:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5269 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4f28 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201370.836069:0:16412:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5269 lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4f28 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788201370.836077:0:16412:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x404:0x0](00000000f3d86d2f) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5269 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4f28 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201370.836140:0:16412:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x404:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:2.0:1788201370.836161:0:16412:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5269 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d4f28 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201370.836169:0:16412:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5269 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d4f28 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1788201370.836189:0:16412:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1788201370.836292:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442715648:192.168.201.130@tcp:2:chmod.0 00000100:00100000:2.0:1788201370.836311:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442715392:192.168.201.130@tcp:2:chmod.0 00000100:00100000:2.0:1788201370.836326:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442715776:192.168.201.130@tcp:2:chmod.0 00000100:00100000:2.0:1788201370.836333:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201370.836347:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442716160:192.168.201.130@tcp:2:chmod.0 00000100:00100000:3.0:1788201370.836752:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442716032:192.168.201.130@tcp:2:chmod.0 00000080:00200000:3.0:1788201370.836799:0:16415:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x407:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201370.836821:0:16415:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-5:[0x200000bd1:0x407:0x0](ffff9023449b4b08) current size 0, valid attrs 10000041, mode 8180, uid 1204961280, gid -28637, new size 0, atime 1788201326.000000000, mtime 1788201326.000000000, ctime 1788201326.000000000, rc 0 00000080:00200000:3.0:1788201370.837354:0:16417:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x409:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201370.837374:0:16417:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-7:[0x200000bd1:0x409:0x0](ffff9023449adb88) current size 0, valid attrs 10000041, mode 8180, uid 1117585408, gid -28637, new size 0, atime 1788201326.000000000, mtime 1788201326.000000000, ctime 1788201326.000000000, rc 0 00000100:00100000:2.0:1788201370.838057:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442715904:192.168.201.130@tcp:2:chmod.0 00000080:00200000:2.0:1788201370.840376:0:16411:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x403:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201370.840409:0:16411:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-1:[0x200000bd1:0x403:0x0](ffff90234497ec08) current size 0, valid attrs 10000041, mode 8180, uid 1117601792, gid -28637, new size 0, atime 1788201326.000000000, mtime 1788201326.000000000, ctime 1788201326.000000000, rc 0 00000080:00200000:3.0:1788201370.841317:0:16414:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x406:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201370.841341:0:16414:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-4:[0x200000bd1:0x406:0x0](ffff902344979148) current size 0, valid attrs 10000041, mode 8180, uid 1094287360, gid -28637, new size 0, atime 1788201326.000000000, mtime 1788201326.000000000, ctime 1788201326.000000000, rc 0 00000100:00100000:1.0:1788201370.843650:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442715520:192.168.201.130@tcp:2:chmod.0 00000080:00200000:1.0:1788201370.845716:0:16416:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x408:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201370.845756:0:16416:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-6:[0x200000bd1:0x408:0x0](ffff9023449ab248) current size 0, valid attrs 10000041, mode 8180, uid 1204961280, gid -28637, new size 0, atime 1788201326.000000000, mtime 1788201326.000000000, ctime 1788201326.000000000, rc 0 00000100:00100000:2.0:1788201370.851501:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442716160:192.168.201.130@tcp:2:chmod.0 00000080:00200000:2.0:1788201370.851571:0:16412:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x404:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201370.851619:0:16412:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-2:[0x200000bd1:0x404:0x0](ffff9023449a7448) current size 0, valid attrs 10000041, mode 8180, uid 1117585408, gid -28637, new size 0, atime 1788201326.000000000, mtime 1788201326.000000000, ctime 1788201326.000000000, rc 0 00000080:00200000:3.0:1788201370.858330:0:16413:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x405:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788201370.858357:0:16413:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file file-3:[0x200000bd1:0x405:0x0](ffff9023449b5348) current size 0, valid attrs 10000041, mode 8180, uid 1204973568, gid -28637, new size 0, atime 1788201326.000000000, mtime 1788201326.000000000, ctime 1788201326.000000000, rc 0 00010000:00010000:0.0:1788201371.265079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201372.295606:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201373.314026:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201374.339976:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201375.367935:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201375.618491:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442716416:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201375.618742:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442716288:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201375.618816:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442716672:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201375.618964:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442716544:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201375.640102:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442716288:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201375.640123:0:2359:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0000-osc-ffff902350a11000: committing for last_committed 4294968871 gen 2 00000100:00100000:0.0:1788201375.640149:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442716544:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201375.640222:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442716416:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201375.640234:0:2360:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff902350a11000: committing for last_committed 4294968873 gen 2 00000100:00100000:1.0:1788201375.640350:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442716672:192.168.201.130@tcp:400:kworker.0 00010000:00010000:3.0:1788201376.385974:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201376.635154:0:17023:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 00010000:00010000:3.0:1788201377.413012:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201378.321858:0:17170:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:2.0:1788201378.433293:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201379.457120:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201380.483146:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788201380.508737:0:17208:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201380.508753:0:17208:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201380.509216:0:17208:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201380.509235:0:17208:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1788201380.509250:0:17208:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:2.0:1788201380.509257:0:17208:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201380.509266:0:17208:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201380.509269:0:17208:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:2.0:1788201380.509276:0:17208:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b180d4bcb8 00010000:00010000:2.0:1788201380.509280:0:17208:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1788201380.509289:0:17208:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200008:2.0:1788201380.512672:0:17208:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788201380.512688:0:17208:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201380.512711:0:17208:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201380.512729:0:17208:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:2.0:1788201380.512746:0:17208:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:2.0:1788201380.512754:0:17208:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201380.512773:0:17208:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201380.512777:0:17208:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:2.0:1788201380.512786:0:17208:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b180d4bcb8 00010000:00010000:2.0:1788201380.512790:0:17208:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:2.0:1788201380.512800:0:17208:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:2.0:1788201380.513227:0:17208:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201380.513238:0:17208:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201380.513246:0:17208:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:2.0:1788201380.513256:0:17208:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:2.0:1788201380.513263:0:17208:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201380.513268:0:17208:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201380.513271:0:17208:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:2.0:1788201380.513275:0:17208:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b180d4bbb0 00010000:00010000:2.0:1788201380.513278:0:17208:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:2.0:1788201380.513285:0:17208:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:2.0:1788201380.513294:0:17208:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 44 00000080:00200008:2.0:1788201380.513302:0:17208:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00000080:00200080:2.0:1788201380.513493:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:2.0:1788201380.513510:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902342dfb400 uarg=0000000000000000 00800000:00000004:2.0:1788201380.513517:0:17208:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: age 647, max_age 656 00000100:00100000:2.0:1788201380.513571:0:17208:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518d180 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17208:1875064442716800:192.168.201.130@tcp:41:lfs.0 00000100:00100000:2.0:1788201380.514009:0:17208:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352e83d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.520306:0:17208:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518d180 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17208:1875064442716800:192.168.201.130@tcp:41:lfs.0 00800000:00000004:0.0:1788201380.520335:0:17208:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: update ffff902350be5190 cache blocks 17239/17271 objects 531669/532010 00000080:00200080:0.0:1788201380.520401:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520419:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520438:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520444:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520448:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520452:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520456:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520459:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520463:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520466:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520470:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520474:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520477:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520481:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520484:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520487:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520491:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520495:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520498:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520502:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520506:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520510:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520513:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520517:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520521:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520524:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520528:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520531:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520535:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520538:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520541:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520544:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520548:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520552:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520556:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520559:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520563:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520566:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520569:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520572:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520575:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520578:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520595:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520600:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520604:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520607:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520611:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520614:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520618:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520621:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520624:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520628:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520631:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520634:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520638:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520641:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520645:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520648:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520651:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520654:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520658:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520661:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520665:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520668:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520672:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520675:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520678:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520682:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520686:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520689:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520693:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520696:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520700:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520704:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520707:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520711:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520715:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520718:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520721:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520724:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520728:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520732:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520735:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520739:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520743:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520746:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520750:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520753:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520756:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520760:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520773:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520777:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520782:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520785:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520789:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520793:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520796:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520800:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520804:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520808:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520812:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520815:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520819:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520821:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520825:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520828:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520832:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520835:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520840:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520844:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520848:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520852:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520856:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520859:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520863:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520866:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520871:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520905:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520911:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520915:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520919:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520923:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520927:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520930:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520934:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520938:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520941:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00800000:00000080:0.0:1788201380.520945:0:17208:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.520949:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.520955:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00020000:00000004:0.0:1788201380.520964:0:17208:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: age 647, max_age 656 00000100:00100000:0.0:1788201380.521001:0:17208:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17208:1875064442716928:192.168.201.130@tcp:13:lfs.0 00000100:00100000:0.0:1788201380.521526:0:17208:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fac00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.525340:0:17208:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17208:1875064442716928:192.168.201.130@tcp:13:lfs.0 00020000:00000004:0.0:1788201380.525369:0:17208:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: update ffff902350be1e10 cache blocks 3667/3683 objects 117728/118668 00000080:00200080:0.0:1788201380.525446:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.525462:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00020000:00000004:0.0:1788201380.525468:0:17208:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: age 647, max_age 656 00000100:00100000:0.0:1788201380.525500:0:17208:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17208:1875064442717056:192.168.201.130@tcp:13:lfs.0 00000100:00100000:0.0:1788201380.525572:0:17208:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fac00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.534526:0:17208:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17208:1875064442717056:192.168.201.130@tcp:13:lfs.0 00020000:00000004:0.0:1788201380.534554:0:17208:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: update ffff902351d65190 cache blocks 3666/3683 objects 117792/118189 00000080:00200080:0.0:1788201380.534754:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534778:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534786:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534790:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534795:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534799:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534803:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534806:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534811:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534814:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534818:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534822:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534826:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534829:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534833:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534836:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534840:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534843:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534847:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534851:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534855:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534859:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534863:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534867:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534872:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534908:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534915:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534919:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534923:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534926:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534931:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534934:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534938:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534941:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534945:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534949:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534953:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534956:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534961:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534964:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534968:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534972:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534976:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534979:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534983:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534987:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534991:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.534994:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.534998:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535002:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535006:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535018:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535023:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535027:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535031:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535034:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535038:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535042:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535046:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535049:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535053:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535056:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535060:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535064:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535068:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535071:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535075:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535079:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535083:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535086:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535090:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535094:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535098:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535101:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535105:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535108:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535112:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535115:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535118:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535122:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535126:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535129:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535133:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535137:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535141:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535144:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535149:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535152:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535156:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535160:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535164:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535168:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535172:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535175:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535179:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535183:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535187:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535190:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535195:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535199:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535202:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535206:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535210:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535213:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535217:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535221:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535225:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535228:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535232:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535235:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535239:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535243:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535246:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535249:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535253:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535257:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535261:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535265:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535269:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535273:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535276:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535280:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535284:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535288:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200080:0.0:1788201380.535292:0:17208:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe1c9a1150 00020000:00000080:0.0:1788201380.535296:0:17208:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90235262fc00 uarg=0000000000000000 00000080:00200008:0.0:1788201380.535310:0:17208:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00000080:00200000:2.0:1788201380.616752:0:17229:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201380.616779:0:17229:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201380.616796:0:17229:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:2.0:1788201380.616812:0:17229:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:2.0:1788201380.616819:0:17229:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201380.616826:0:17229:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201380.616830:0:17229:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:2.0:1788201380.616836:0:17229:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181cf3b58 00010000:00010000:2.0:1788201380.616841:0:17229:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:2.0:1788201380.616849:0:17229:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:2.0:1788201380.616865:0:17229:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201380.616900:0:17229:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d102d.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=24 00000080:00200000:2.0:1788201380.616904:0:17229:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d102d.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=0 00000002:00010000:2.0:1788201380.616916:0:17229:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d102d.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1788201380.616963:0:17229:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17229 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.616972:0:17229:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17229 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.616982:0:17229:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17229 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201380.617007:0:17229:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518d880 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17229:1875064442717184:192.168.201.130@tcp:101:checkstat.0 00000100:00100000:2.0:1788201380.617188:0:17229:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352e83600 going to sleep for 11 seconds 00000100:00100000:3.0:1788201380.633936:0:17229:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518d880 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17229:1875064442717184:192.168.201.130@tcp:101:checkstat.0 00010000:00010000:3.0:1788201380.633963:0:17229:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.633985:0:17229:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.633994:0:17229:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.634005:0:17229:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.634016:0:17229:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.634030:0:17229:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.634039:0:17229:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.634052:0:17229:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201380.634066:0:17229:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff90234518d880 x1875064442717184/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201396 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201380.634079:0:17229:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.634089:0:17229:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1026 (0 0) 00000080:00200000:3.0:1788201380.634097:0:17229:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201380.634119:0:17229:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x402:0x0], updating i_size 7680 00000080:00200000:3.0:1788201380.634122:0:17229:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x402:0x0](ffff9023449a1988): rc = 0 00000080:00010000:3.0:1788201380.634128:0:17229:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x402:0x0](ffff9023449a1988) for lock 0x4605c7c97e1c529a 00000080:00010000:3.0:1788201380.634141:0:17229:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x402:0x0](ffff9023449a1988) 00000080:00010000:3.0:1788201380.634145:0:17229:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c529a from it ffffa0b181cf3a28 00010000:00010000:3.0:1788201380.634148:0:17229:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.634155:0:17229:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201380.634184:0:17229:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201380.634201:0:17229:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-1, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), flags=8 00000080:00200000:3.0:1788201380.634205:0:17229:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-1, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=getattr 00000002:00010000:3.0:1788201380.634221:0:17229:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-1,[0x0:0x0:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201380.634259:0:17229:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17229 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.634267:0:17229:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17229 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.634277:0:17229:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17229 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201380.634298:0:17229:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17229:1875064442717312:192.168.201.130@tcp:101:checkstat.0 00000100:00100000:3.0:1788201380.634378:0:17229:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201380.647820:0:17229:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17229:1875064442717312:192.168.201.130@tcp:101:checkstat.0 00010000:00010000:3.0:1788201380.647848:0:17229:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.647865:0:17229:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.647905:0:17229:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.647919:0:17229:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.647930:0:17229:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.647943:0:17229:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.647957:0:17229:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.647965:0:17229:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201380.647977:0:17229:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352689180 x1875064442717312/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201396 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201380.647988:0:17229:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201380.648000:0:17229:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.648009:0:17229:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1027 (0 0) 00000080:00200000:3.0:1788201380.648016:0:17229:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201380.648089:0:17229:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x403:0x0](ffff90234497ec08): rc = 0 00000080:00200000:3.0:1788201380.648109:0:17229:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x403:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201380.648120:0:17229:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x403:0x0](ffff90234497ec08) for lock 0x4605c7c97e1c52a1 00000080:00010000:3.0:1788201380.648131:0:17229:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x403:0x0](ffff90234497ec08) 00000080:00010000:3.0:1788201380.648134:0:17229:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52a1 from it ffffa0b181cf3b20 00010000:00010000:3.0:1788201380.648137:0:17229:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.648147:0:17229:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788201380.648182:0:17229:0:(file.c:6283:ll_getattr_dentry()) START file file-1:[0x200000bd1:0x403:0x0](ffff90234497ec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201380.648189:0:17229:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x403:0x0](ffff90234497ec08),name=file-1 00000002:00010000:3.0:1788201380.648206:0:17229:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x403:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201380.648214:0:17229:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201380.648225:0:17229:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201380.648232:0:17229:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201380.648237:0:17229:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x403:0x0](ffff90234497ec08) 00000080:00010000:3.0:1788201380.648240:0:17229:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x403:0x0](ffff90234497ec08) for lock 0x4605c7c97e1c52a1 00000080:00010000:3.0:1788201380.648244:0:17229:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52a1 from it ffffa0b181cf3cb8 00010000:00010000:3.0:1788201380.648247:0:17229:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201380.648255:0:17229:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201380.648279:0:17229:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x403:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1788201380.648287:0:17229:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x403:0x0] 00000008:00010000:3.0:1788201380.648312:0:17229:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788201380.648316:0:17229:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023483ef7b0, osc lock ffff902348866d10, flags 0x400001002 00010000:00010000:3.0:1788201380.648333:0:17229:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x442:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201380.648351:0:17229:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:3.0:1788201380.648363:0:17229:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788201380.648367:0:17229:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201380.648375:0:17229:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:3.0:1788201380.648387:0:17229:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1788201380.648412:0:17229:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x403:0x0] updating i_size 0 i_blocks 1 00000020:00010000:3.0:1788201380.648416:0:17229:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235000c4a0 (ffff902346343d88/0) at cl_lock_release():245 00000020:00010000:3.0:1788201380.648419:0:17229:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235000c4a0 (ffff902346343d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201380.648430:0:17229:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023483ef7b0 (ffff902346343758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788201380.648435:0:17229:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235000c4a0 (ffff902346343d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201380.648438:0:17229:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023483ef7b0 (ffff902346343d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201380.648456:0:17229:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x403:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201380.657342:0:17232:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201380.657369:0:17232:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201380.657386:0:17232:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:3.0:1788201380.657401:0:17232:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:3.0:1788201380.657415:0:17232:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201380.657432:0:17232:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201380.657436:0:17232:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:3.0:1788201380.657443:0:17232:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181a53b58 00010000:00010000:3.0:1788201380.657447:0:17232:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:3.0:1788201380.657455:0:17232:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:3.0:1788201380.657471:0:17232:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201380.657479:0:17232:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=24 00000080:00200000:3.0:1788201380.657484:0:17232:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201380.657494:0:17232:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-2, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), flags=8 00000080:00200000:3.0:1788201380.657498:0:17232:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-2, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=getattr 00000002:00010000:3.0:1788201380.657509:0:17232:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-2,[0x0:0x0:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201380.657553:0:17232:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17232 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.657560:0:17232:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17232 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.657570:0:17232:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17232 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201380.657609:0:17232:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17232:1875064442717440:192.168.201.130@tcp:101:checkstat.0 00000100:00100000:3.0:1788201380.657704:0:17232:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f579c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.668035:0:17232:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17232:1875064442717440:192.168.201.130@tcp:101:checkstat.0 00010000:00010000:0.0:1788201380.668062:0:17232:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.668079:0:17232:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.668087:0:17232:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.668099:0:17232:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.668110:0:17232:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.668123:0:17232:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.668130:0:17232:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.668137:0:17232:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201380.668153:0:17232:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352689180 x1875064442717440/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201396 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201380.668164:0:17232:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201380.668176:0:17232:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.668185:0:17232:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1028 (0 0) 00000080:00200000:0.0:1788201380.668191:0:17232:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201380.668269:0:17232:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x404:0x0](ffff9023449a7448): rc = 0 00000080:00200000:0.0:1788201380.668288:0:17232:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x404:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201380.668298:0:17232:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x404:0x0](ffff9023449a7448) for lock 0x4605c7c97e1c52a8 00000080:00010000:0.0:1788201380.668307:0:17232:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x404:0x0](ffff9023449a7448) 00000080:00010000:0.0:1788201380.668311:0:17232:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52a8 from it ffffa0b181a53b20 00010000:00010000:0.0:1788201380.668315:0:17232:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.668323:0:17232:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201380.668354:0:17232:0:(file.c:6283:ll_getattr_dentry()) START file file-2:[0x200000bd1:0x404:0x0](ffff9023449a7448), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201380.668365:0:17232:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x404:0x0](ffff9023449a7448),name=file-2 00000002:00010000:0.0:1788201380.668381:0:17232:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x404:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201380.668390:0:17232:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.668401:0:17232:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201380.668408:0:17232:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201380.668413:0:17232:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x404:0x0](ffff9023449a7448) 00000080:00010000:0.0:1788201380.668416:0:17232:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x404:0x0](ffff9023449a7448) for lock 0x4605c7c97e1c52a8 00000080:00010000:0.0:1788201380.668419:0:17232:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52a8 from it ffffa0b181a53cb8 00010000:00010000:0.0:1788201380.668438:0:17232:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.668447:0:17232:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.668472:0:17232:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x404:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201380.668481:0:17232:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x404:0x0] 00000008:00010000:0.0:1788201380.668509:0:17232:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201380.668513:0:17232:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff90235306dbb0, osc lock ffff902343fb7a20, flags 0x400001002 00010000:00010000:0.0:1788201380.668530:0:17232:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x442:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.668541:0:17232:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1788201380.668553:0:17232:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788201380.668558:0:17232:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.668570:0:17232:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1788201380.668581:0:17232:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788201380.669989:0:17232:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x404:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201380.670001:0:17232:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346370000 (ffff902348a2f368/0) at cl_lock_release():245 00000020:00010000:0.0:1788201380.670006:0:17232:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346370000 (ffff902348a2f368/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201380.670011:0:17232:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235306dbb0 (ffff902348a2f128/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201380.670015:0:17232:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346370000 (ffff902348a2f368/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201380.670018:0:17232:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235306dbb0 (ffff902348a2f368/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201380.670055:0:17232:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x404:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201380.677845:0:17235:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201380.677899:0:17235:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201380.677919:0:17235:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:0.0:1788201380.677935:0:17235:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:0.0:1788201380.677943:0:17235:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201380.677950:0:17235:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201380.677954:0:17235:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:0.0:1788201380.677961:0:17235:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b180d7fb58 00010000:00010000:0.0:1788201380.677965:0:17235:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:0.0:1788201380.677979:0:17235:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:0.0:1788201380.677997:0:17235:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201380.678004:0:17235:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=24 00000080:00200000:0.0:1788201380.678010:0:17235:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201380.678019:0:17235:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-3, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), flags=8 00000080:00200000:0.0:1788201380.678023:0:17235:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-3, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=getattr 00000002:00010000:0.0:1788201380.678036:0:17235:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-3,[0x0:0x0:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201380.678079:0:17235:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17235 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.678088:0:17235:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17235 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.678098:0:17235:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17235 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201380.678122:0:17235:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17235:1875064442717568:192.168.201.130@tcp:101:checkstat.0 00000100:00100000:0.0:1788201380.678208:0:17235:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fac00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.689197:0:17235:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17235:1875064442717568:192.168.201.130@tcp:101:checkstat.0 00010000:00010000:0.0:1788201380.689227:0:17235:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.689248:0:17235:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.689261:0:17235:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.689272:0:17235:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.689283:0:17235:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.689296:0:17235:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.689305:0:17235:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.689312:0:17235:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201380.689324:0:17235:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023488b1880 x1875064442717568/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201396 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201380.689336:0:17235:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201380.689349:0:17235:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.689364:0:17235:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1029 (0 0) 00000080:00200000:0.0:1788201380.689371:0:17235:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201380.689465:0:17235:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x405:0x0](ffff9023449b5348): rc = 0 00000080:00200000:0.0:1788201380.689488:0:17235:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x405:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201380.689499:0:17235:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x405:0x0](ffff9023449b5348) for lock 0x4605c7c97e1c52af 00000080:00010000:0.0:1788201380.689511:0:17235:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x405:0x0](ffff9023449b5348) 00000080:00010000:0.0:1788201380.689515:0:17235:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52af from it ffffa0b180d7fb20 00010000:00010000:0.0:1788201380.689519:0:17235:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.689528:0:17235:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201380.689567:0:17235:0:(file.c:6283:ll_getattr_dentry()) START file file-3:[0x200000bd1:0x405:0x0](ffff9023449b5348), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201380.690021:0:17235:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x405:0x0](ffff9023449b5348),name=file-3 00000002:00010000:0.0:1788201380.690042:0:17235:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x405:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201380.690057:0:17235:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.690072:0:17235:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201380.690079:0:17235:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201380.690085:0:17235:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x405:0x0](ffff9023449b5348) 00000080:00010000:0.0:1788201380.690088:0:17235:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x405:0x0](ffff9023449b5348) for lock 0x4605c7c97e1c52af 00000080:00010000:0.0:1788201380.690093:0:17235:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52af from it ffffa0b180d7fcb8 00010000:00010000:0.0:1788201380.690096:0:17235:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.690104:0:17235:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.690134:0:17235:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x405:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201380.690142:0:17235:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x405:0x0] 00000008:00010000:0.0:1788201380.690172:0:17235:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201380.690175:0:17235:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff90235306dbb0, osc lock ffff902343fb7a20, flags 0x400001002 00010000:00010000:0.0:1788201380.690192:0:17235:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x443:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.690205:0:17235:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1788201380.690218:0:17235:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788201380.690222:0:17235:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.690231:0:17235:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1788201380.690241:0:17235:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788201380.690253:0:17235:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x405:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201380.690257:0:17235:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346370000 (ffff902348a2f368/0) at cl_lock_release():245 00000020:00010000:0.0:1788201380.690260:0:17235:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346370000 (ffff902348a2f368/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201380.690264:0:17235:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235306dbb0 (ffff902348a2f128/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201380.690268:0:17235:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346370000 (ffff902348a2f368/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201380.690271:0:17235:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235306dbb0 (ffff902348a2f368/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201380.690292:0:17235:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x405:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201380.694631:0:17240:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201380.694657:0:17240:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201380.694674:0:17240:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:2.0:1788201380.694689:0:17240:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:2.0:1788201380.694703:0:17240:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201380.694710:0:17240:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201380.694714:0:17240:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:2.0:1788201380.694721:0:17240:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b180d7fb58 00010000:00010000:2.0:1788201380.694725:0:17240:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:2.0:1788201380.694734:0:17240:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:2.0:1788201380.694750:0:17240:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201380.694757:0:17240:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=24 00000080:00200000:2.0:1788201380.694762:0:17240:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201380.694771:0:17240:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-4, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), flags=8 00000080:00200000:2.0:1788201380.694775:0:17240:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-4, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=getattr 00010000:00010000:3.0:1788201380.696611:0:17238:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.696624:0:17238:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.696647:0:17238:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.696653:0:17238:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.696668:0:17238:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.696675:0:17238:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.696690:0:17238:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 6 type: IBT flags: 0x10000000000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.696697:0:17238:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.696712:0:17238:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c52d2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 7 type: IBT flags: 0x10000000000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.696718:0:17238:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c52d2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x0 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.696732:0:17238:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c52d9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 8 type: IBT flags: 0x10000000000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.696742:0:17238:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c52d9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x0 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.696759:0:17238:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c52e0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 9 type: IBT flags: 0x10000000000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.696766:0:17238:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c52e0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x0 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.696781:0:17238:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c52e7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 10 type: IBT flags: 0x10000000000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.696787:0:17238:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c52e7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x0 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201380.696856:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442717696:192.168.201.130@tcp:63:ll_sa.0 00000002:00010000:2.0:1788201380.698329:0:17240:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-4,[0x0:0x0:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201380.698378:0:17240:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 11 type: IBT flags: 0x10000000000000 pid: 17240 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.698389:0:17240:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 11 type: IBT flags: 0x0 pid: 17240 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.698400:0:17240:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 11 type: IBT flags: 0x0 pid: 17240 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201380.698420:0:17240:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518df80 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17240:1875064442717824:192.168.201.130@tcp:101:checkstat.0 00010000:00010000:0.0:1788201380.698665:0:17241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 12 type: IBT flags: 0x10000000000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.698674:0:17241:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x0 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.698693:0:17241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 13 type: IBT flags: 0x10000000000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.698700:0:17241:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x0 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.698714:0:17241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 14 type: IBT flags: 0x10000000000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.698722:0:17241:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x0 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.698737:0:17241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c530a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 15 type: IBT flags: 0x10000000000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.698744:0:17241:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c530a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT flags: 0x0 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.698759:0:17241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5311 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 16 type: IBT flags: 0x10000000000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.698765:0:17241:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5311 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x0 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.698785:0:17241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c5318 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 17 type: IBT flags: 0x10000000000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.698792:0:17241:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c5318 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT flags: 0x0 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.698807:0:17241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c531f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 18 type: IBT flags: 0x10000000000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.698814:0:17241:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c531f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT flags: 0x0 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.698830:0:17241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c5326 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 19 type: IBT flags: 0x10000000000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.698837:0:17241:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c5326 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 19 type: IBT flags: 0x0 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201380.698902:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442717952:192.168.201.130@tcp:63:ll_sa.0 00000100:00100000:2.0:1788201380.699003:0:17240:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352e83600 going to sleep for 11 seconds 00000100:00100000:2.0:1788201380.712110:0:17240:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518df80 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17240:1875064442717824:192.168.201.130@tcp:101:checkstat.0 00010000:00010000:2.0:1788201380.712136:0:17240:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 19 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.712154:0:17240:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.712162:0:17240:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.712174:0:17240:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.712185:0:17240:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.712198:0:17240:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.712206:0:17240:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.712215:0:17240:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201380.712227:0:17240:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff90234518df80 x1875064442717824/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201396 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201380.712239:0:17240:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201380.712253:0:17240:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.712263:0:17240:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1030 (0 0) 00000080:00200000:2.0:1788201380.712270:0:17240:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201380.712355:0:17240:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x406:0x0](ffff902344979148): rc = 0 00000080:00200000:2.0:1788201380.712379:0:17240:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x406:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1788201380.712390:0:17240:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x406:0x0](ffff902344979148) for lock 0x4605c7c97e1c52ee 00000080:00010000:2.0:1788201380.712400:0:17240:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x406:0x0](ffff902344979148) 00000080:00010000:2.0:1788201380.712404:0:17240:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52ee from it ffffa0b180d7fb20 00010000:00010000:2.0:1788201380.712408:0:17240:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.712416:0:17240:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788201380.712458:0:17240:0:(file.c:6283:ll_getattr_dentry()) START file file-4:[0x200000bd1:0x406:0x0](ffff902344979148), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201380.712464:0:17240:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x406:0x0](ffff902344979148),name=file-4 00000002:00010000:2.0:1788201380.712481:0:17240:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x406:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201380.712490:0:17240:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201380.712500:0:17240:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201380.712507:0:17240:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201380.712513:0:17240:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x406:0x0](ffff902344979148) 00000080:00010000:2.0:1788201380.712515:0:17240:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x406:0x0](ffff902344979148) for lock 0x4605c7c97e1c52ee 00000080:00010000:2.0:1788201380.712519:0:17240:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52ee from it ffffa0b180d7fcb8 00010000:00010000:2.0:1788201380.712522:0:17240:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201380.712528:0:17240:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201380.712552:0:17240:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x406:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201380.712572:0:17240:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x406:0x0] 00000008:00010000:2.0:1788201380.712599:0:17240:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201380.712604:0:17240:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398530, osc lock ffff9023500dce40, flags 0x400001002 00010000:00010000:2.0:1788201380.712621:0:17240:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x443:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201380.712633:0:17240:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788201380.712646:0:17240:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788201380.712650:0:17240:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201380.712658:0:17240:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1788201380.712669:0:17240:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788201380.712681:0:17240:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x406:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201380.712685:0:17240:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201380.712688:0:17240:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201380.712691:0:17240:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398530 (ffff902350b9de18/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201380.712706:0:17240:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201380.712709:0:17240:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398530 (ffff902350b9d7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201380.712729:0:17240:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x406:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1788201380.715478:0:2362:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d505c expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715504:0:2362:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d505c expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715511:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d505c expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715521:0:2362:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d505c expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715537:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d505c expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715550:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d505c expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715558:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d505c expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715566:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d505c expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201380.715581:0:2362:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff90235268b100 x1875064442717696/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3408/7976 e 0 to 0 dl 1788201396 ref 1 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1788201380.715593:0:2362:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d505c expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1788201380.715608:0:2362:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52b6 from it ffff9023584a6190 00010000:00010000:3.0:1788201380.715611:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d505c expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715619:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d505c expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201380.715646:0:2362:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715660:0:2362:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715666:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715673:0:2362:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715681:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715689:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715695:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715700:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201380.715708:0:2362:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff90235268b100 x1875064442717696/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3408/7976 e 0 to 0 dl 1788201396 ref 2 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1788201380.715716:0:2362:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1788201380.715728:0:2362:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52bd from it ffff9023584a4d90 00010000:00010000:3.0:1788201380.715731:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715737:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201380.715749:0:2362:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715757:0:2362:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715763:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715769:0:2362:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715775:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715782:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715787:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715797:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201380.715805:0:2362:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff90235268b100 x1875064442717696/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3408/7976 e 0 to 0 dl 1788201396 ref 3 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1788201380.715811:0:2362:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1788201380.715818:0:2362:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52c4 from it ffff9023584a5590 00010000:00010000:3.0:1788201380.715820:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715827:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201380.715837:0:2362:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715844:0:2362:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715850:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715856:0:2362:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715866:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715872:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715895:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715901:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201380.715909:0:2362:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff90235268b100 x1875064442717696/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3408/7976 e 0 to 0 dl 1788201396 ref 4 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:3.0:1788201380.715916:0:2362:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1788201380.715923:0:2362:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52cb from it ffff9023584a7990 00010000:00010000:3.0:1788201380.715926:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715932:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201380.715942:0:2362:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c52d2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x0 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715952:0:2362:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c52d2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715957:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c52d2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x10206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715962:0:2362:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c52d2 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x10206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715968:0:2362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c52d2 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715974:0:2362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db0400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201380.715979:0:2362:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c52d2 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 14 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715985:0:2362:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c52d2 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 14 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715990:0:2362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c52d2 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 14 type: IBT flags: 0x20e400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.715997:0:2362:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201380.716002:0:2362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c52d2 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 14 type: IBT flags: 0x4a0f400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718093:0:2362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0400/0x4605c7c97e1c52d2 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 14 type: IBT flags: 0x4a0f400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718124:0:2362:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c52d9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x0 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718130:0:2362:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c52d9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718135:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c52d9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x10206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718142:0:2362:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c52d9 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x10206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718148:0:2362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c52d9 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718153:0:2362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db1600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201380.718157:0:2362:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c52d9 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718163:0:2362:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c52d9 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718169:0:2362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c52d9 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x20e400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718175:0:2362:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201380.718180:0:2362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c52d9 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x4a0f400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718185:0:2362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1600/0x4605c7c97e1c52d9 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x4a0f400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718195:0:2362:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c52e0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x0 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718200:0:2362:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c52e0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718205:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c52e0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x10206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718209:0:2362:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c52e0 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x10206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718214:0:2362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c52e0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718218:0:2362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342db0600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201380.718220:0:2362:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c52e0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 12 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718225:0:2362:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c52e0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 12 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718230:0:2362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c52e0 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 12 type: IBT flags: 0x20e400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718234:0:2362:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201380.718237:0:2362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c52e0 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 12 type: IBT flags: 0x4a0f400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718242:0:2362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0600/0x4605c7c97e1c52e0 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 12 type: IBT flags: 0x4a0f400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718251:0:2362:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c52e7 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x0 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718256:0:2362:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c52e7 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718261:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c52e7 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x10206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718265:0:2362:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c52e7 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x10206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718270:0:2362:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c52e7 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718274:0:2362:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351684c00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201380.718277:0:2362:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c52e7 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718281:0:2362:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c52e7 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718286:0:2362:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c52e7 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x20e400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718290:0:2362:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201380.718292:0:2362:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c52e7 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x4a0f400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.718297:0:2362:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c52e7 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x4a0f400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201380.718314:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442717696:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:0.0:1788201380.718818:0:2359:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50fd expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.718835:0:2359:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50fd expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.718844:0:2359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50fd expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.718854:0:2359:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d50fd expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.718865:0:2359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50fd expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.718871:0:2359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d50fd expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.718901:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d50fd expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.718906:0:2359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d50fd expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201380.718913:0:2359:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023488b1880 x1875064442717952/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3408/7976 e 0 to 0 dl 1788201396 ref 1 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:0.0:1788201380.718921:0:2359:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50fd expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:0.0:1788201380.718929:0:2359:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52f5 from it ffff90235262fd90 00010000:00010000:0.0:1788201380.718932:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d50fd expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.718937:0:2359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d50fd expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201380.718955:0:765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d505c expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.718956:0:2359:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5119 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.718981:0:2359:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5119 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.718986:0:2359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5119 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.718990:0:2359:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5119 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.718995:0:2359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5119 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.718999:0:2359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5119 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719003:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5119 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719007:0:2359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5119 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201380.719013:0:2359:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023488b1880 x1875064442717952/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3408/7976 e 0 to 0 dl 1788201396 ref 2 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:0.0:1788201380.719018:0:2359:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5119 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:0.0:1788201380.719027:0:2359:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52fc from it ffff90235262f990 00010000:00010000:0.0:1788201380.719029:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5119 expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719033:0:2359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5119 expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.719042:0:2359:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5135 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719047:0:2359:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5135 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719052:0:2359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5135 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719056:0:2359:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5135 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719061:0:2359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5135 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1788201380.719063:0:765:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x406:0x0](ffff902344979148): rc = 0 00010000:00010000:0.0:1788201380.719066:0:2359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5135 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719072:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5135 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719080:0:2359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5135 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1788201380.719082:0:765:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x406:0x0]: layout version change: 0 -> 0 00000002:00100000:0.0:1788201380.719087:0:2359:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023488b1880 x1875064442717952/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 3408/7976 e 0 to 0 dl 1788201396 ref 3 fl Interpret:RQU/200/0 rc -2/-2 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000002:00010000:0.0:1788201380.719093:0:2359:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5135 expref: -99 pid: 17241 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1788201380.719094:0:765:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x406:0x0](ffff902344979148) for lock 0x4605c7c97e1c52b6 00000080:00010000:0.0:1788201380.719101:0:2359:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5303 from it ffff90235262c990 00010000:00010000:0.0:1788201380.719105:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5135 expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719109:0:2359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5135 expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201380.719113:0:765:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52b6 from it ffff9023584a6190 00010000:00010000:0.0:1788201380.719118:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c530a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x0 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.719119:0:765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d505c expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.719131:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c530a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.719136:0:765:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d505c expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.719137:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c530a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x10206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719150:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c530a lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x10206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719159:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c530a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719163:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902343730600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201380.719165:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c530a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719169:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c530a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719173:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c530a lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x20e400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.719173:0:765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.719186:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201380.719189:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c530a lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x4a0f400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719194:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c530a lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x4a0f400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719203:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5311 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719207:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5311 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719211:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5311 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719214:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5311 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1788201380.719218:0:765:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x407:0x0](ffff9023449b4b08): rc = 0 00010000:00010000:0.0:1788201380.719219:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5311 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719226:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902343730400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201380.719228:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5311 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719231:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5311 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719235:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5311 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x20e400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1788201380.719236:0:765:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x407:0x0]: layout version change: 0 -> 0 00010000:00010000:0.0:1788201380.719240:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201380.719244:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5311 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719248:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5311 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1788201380.719248:0:765:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x407:0x0](ffff9023449b4b08) for lock 0x4605c7c97e1c52bd 00000080:00010000:3.0:1788201380.719258:0:765:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52bd from it ffff9023584a4d90 00010000:00010000:0.0:1788201380.719263:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c5318 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.719263:0:765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.719279:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c5318 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.719281:0:765:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.719292:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c5318 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719300:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c5318 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.719306:0:765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.719309:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c5318 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719323:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902343731200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201380.719327:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c5318 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719331:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c5318 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719334:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c5318 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x20e400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719339:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201380.719343:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c5318 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719349:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c5318 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1788201380.719352:0:765:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x408:0x0](ffff9023449ab248): rc = 0 00010000:00010000:0.0:1788201380.719363:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c531f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1788201380.719369:0:765:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x408:0x0]: layout version change: 0 -> 0 00010000:00010000:0.0:1788201380.719371:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c531f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719377:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c531f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719383:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c531f lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719391:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c531f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719396:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902343730000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201380.719400:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c531f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1788201380.719400:0:765:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x408:0x0](ffff9023449ab248) for lock 0x4605c7c97e1c52c4 00000080:00010000:3.0:1788201380.719410:0:765:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52c4 from it ffff9023584a5590 00010000:00010000:0.0:1788201380.719412:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c531f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.719416:0:765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.719432:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c531f lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x20e400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.719435:0:765:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.719446:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201380.719452:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c531f lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719459:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c531f lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.719462:0:765:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.719476:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c5326 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719482:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c5326 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719485:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c5326 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719489:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c5326 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719494:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c5326 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719498:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902343731600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201380.719501:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c5326 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1788201380.719501:0:765:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x409:0x0](ffff9023449adb88): rc = 0 00010000:00010000:0.0:1788201380.719509:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c5326 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719513:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c5326 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x20e400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719517:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000080:00200000:3.0:1788201380.719518:0:765:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x409:0x0]: layout version change: 0 -> 0 00010000:00010000:0.0:1788201380.719520:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c5326 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.719524:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c5326 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00010000:3.0:1788201380.719528:0:765:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x409:0x0](ffff9023449adb88) for lock 0x4605c7c97e1c52cb 00000080:00010000:3.0:1788201380.719534:0:765:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52cb from it ffff9023584a7990 00000100:00100000:0.0:1788201380.719535:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442717952:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:3.0:1788201380.719540:0:765:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201380.719549:0:765:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.719555:0:16782:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d50fd expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201380.719611:0:16782:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x407:0x0](ffff9023449b4b08): rc = 0 00000080:00200000:0.0:1788201380.719627:0:16782:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x407:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201380.719636:0:16782:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x407:0x0](ffff9023449b4b08) for lock 0x4605c7c97e1c52f5 00000080:00010000:0.0:1788201380.719640:0:16782:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52f5 from it ffff90235262fd90 00010000:00010000:0.0:1788201380.719644:0:16782:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d50fd expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.719652:0:16782:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d50fd expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.719667:0:16782:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5119 expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201380.719703:0:16782:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x408:0x0](ffff9023449ab248): rc = 0 00000080:00200000:0.0:1788201380.719728:0:16782:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x408:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201380.719741:0:16782:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x408:0x0](ffff9023449ab248) for lock 0x4605c7c97e1c52fc 00000080:00010000:0.0:1788201380.719745:0:16782:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52fc from it ffff90235262f990 00010000:00010000:0.0:1788201380.719747:0:16782:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5119 expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.719753:0:16782:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5119 expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.719765:0:16782:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5135 expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201380.719791:0:16782:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x409:0x0](ffff9023449adb88): rc = 0 00000080:00200000:0.0:1788201380.719812:0:16782:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x409:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201380.719824:0:16782:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x409:0x0](ffff9023449adb88) for lock 0x4605c7c97e1c5303 00000080:00010000:0.0:1788201380.719827:0:16782:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5303 from it ffff90235262c990 00010000:00010000:0.0:1788201380.719831:0:16782:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5135 expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.719844:0:16782:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5135 expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201380.722055:0:17239:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x407:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201380.722070:0:17239:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x407:0x0] 00000008:00010000:2.0:1788201380.722101:0:17239:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:2.0:1788201380.722106:0:17239:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398530, osc lock ffff9023500dce40, flags 0x400041012 00010000:00010000:2.0:1788201380.722127:0:17239:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201380.722143:0:17239:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201380.722155:0:17239:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201380.722164:0:17239:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:2.0:1788201380.722181:0:17239:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201380.722184:0:17239:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201380.722187:0:17239:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398530 (ffff902350b9de18/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201380.722191:0:17239:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201380.722194:0:17239:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398530 (ffff902350b9d7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201380.722220:0:17239:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x407:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201380.722246:0:17239:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x408:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201380.722251:0:17239:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x408:0x0] 00000008:00010000:2.0:1788201380.722268:0:17239:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:2.0:1788201380.722270:0:17239:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398530, osc lock ffff9023500dce40, flags 0x400041012 00010000:00010000:2.0:1788201380.722281:0:17239:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201380.722290:0:17239:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201380.722300:0:17239:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201380.722309:0:17239:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:2.0:1788201380.722320:0:17239:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201380.722322:0:17239:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201380.722325:0:17239:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398530 (ffff902350b9de18/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201380.722328:0:17239:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201380.722330:0:17239:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398530 (ffff902350b9d7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201380.722343:0:17239:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x408:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201380.722362:0:17239:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x409:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201380.722366:0:17239:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x409:0x0] 00000008:00010000:2.0:1788201380.722382:0:17239:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:2.0:1788201380.722384:0:17239:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398530, osc lock ffff9023500dce40, flags 0x400041012 00010000:00010000:2.0:1788201380.722392:0:17239:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201380.722402:0:17239:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201380.722410:0:17239:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201380.722417:0:17239:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00010000:2.0:1788201380.722437:0:17239:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201380.722439:0:17239:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201380.722442:0:17239:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398530 (ffff902350b9de18/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201380.722445:0:17239:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201380.722447:0:17239:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398530 (ffff902350b9d7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201380.722459:0:17239:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x409:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201380.733990:0:17244:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201380.734021:0:17244:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201380.734041:0:17244:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:0.0:1788201380.734058:0:17244:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:0.0:1788201380.734066:0:17244:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201380.734075:0:17244:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201380.734078:0:17244:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:0.0:1788201380.734085:0:17244:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b1814fbb58 00010000:00010000:0.0:1788201380.734089:0:17244:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:0.0:1788201380.734098:0:17244:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:0.0:1788201380.734116:0:17244:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201380.734123:0:17244:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=24 00000080:00200000:0.0:1788201380.734128:0:17244:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201380.734138:0:17244:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-5, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), flags=8 00000080:00200000:0.0:1788201380.734142:0:17244:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-5, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=getattr 00000002:00010000:0.0:1788201380.734169:0:17244:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-5,[0x0:0x0:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201380.734218:0:17244:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17244 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.734226:0:17244:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17244 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.734236:0:17244:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17244 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201380.734263:0:17244:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17244:1875064442718080:192.168.201.130@tcp:101:checkstat.0 00000100:00100000:0.0:1788201380.734354:0:17244:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa6c0 going to sleep for 11 seconds 00010000:00010000:3.0:1788201380.734721:0:17238:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c5334 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.734727:0:17238:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c5334 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201380.734892:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442718208:192.168.201.130@tcp:63:ll_sa.0 00000100:00100000:2.0:1788201380.738199:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201380.738225:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442718336:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201380.741370:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442718336:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201380.741405:0:2361:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c5334 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.741431:0:2361:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c5334 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.741438:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c5334 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.741446:0:2361:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c5334 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.741456:0:2361:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c5334 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.741462:0:2361:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351685400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201380.741467:0:2361:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c5334 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.741476:0:2361:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c5334 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.741485:0:2361:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c5334 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x20e400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.741492:0:2361:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201380.741500:0:2361:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c5334 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.741507:0:2361:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c5334 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 17238 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201380.741533:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442718208:192.168.201.130@tcp:63:ll_sa.0 00000100:00100000:0.0:1788201380.741678:0:17244:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17244:1875064442718080:192.168.201.130@tcp:101:checkstat.0 00010000:00010000:0.0:1788201380.741806:0:17244:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.741821:0:17244:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.741829:0:17244:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.741839:0:17244:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.741850:0:17244:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.741863:0:17244:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.741871:0:17244:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.742009:0:17244:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201380.742022:0:17244:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023488b1880 x1875064442718080/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201396 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201380.742034:0:17244:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201380.742048:0:17244:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.742060:0:17244:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.742071:0:17244:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.742080:0:17244:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.742087:0:17244:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.742099:0:17244:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.742153:0:17244:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:0.0:1788201380.742160:0:17244:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201380.742244:0:17244:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x407:0x0](ffff9023449b4b08): rc = 0 00000080:00200000:0.0:1788201380.742266:0:17244:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x407:0x0]: layout version change: 0 -> 0 00000080:00010000:0.0:1788201380.742278:0:17244:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x407:0x0](ffff9023449b4b08) for lock 0x4605c7c97e1c52bd 00000080:00010000:0.0:1788201380.742289:0:17244:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x407:0x0](ffff9023449b4b08) 00000080:00010000:0.0:1788201380.742292:0:17244:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52bd from it ffffa0b1814fbb20 00010000:00010000:0.0:1788201380.742296:0:17244:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.742305:0:17244:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00000080:00200008:0.0:1788201380.742337:0:17244:0:(file.c:6283:ll_getattr_dentry()) START file file-5:[0x200000bd1:0x407:0x0](ffff9023449b4b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201380.742344:0:17244:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x407:0x0](ffff9023449b4b08),name=file-5 00000002:00010000:0.0:1788201380.742360:0:17244:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x407:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201380.742368:0:17244:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788201380.742376:0:17244:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00000080:00200000:0.0:1788201380.742383:0:17244:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201380.742388:0:17244:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x407:0x0](ffff9023449b4b08) 00000080:00010000:0.0:1788201380.742391:0:17244:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x407:0x0](ffff9023449b4b08) for lock 0x4605c7c97e1c52bd 00000080:00010000:0.0:1788201380.742394:0:17244:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52bd from it ffffa0b1814fbcb8 00010000:00010000:0.0:1788201380.742397:0:17244:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788201380.742404:0:17244:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000080:00200000:0.0:1788201380.742439:0:17244:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x407:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201380.742448:0:17244:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x407:0x0] 00000008:00010000:0.0:1788201380.742476:0:17244:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201380.742480:0:17244:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff90235306dfb0, osc lock ffff902343fb7a20, flags 0x400001002 00010000:00010000:0.0:1788201380.742498:0:17244:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.742510:0:17244:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1788201380.742522:0:17244:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788201380.742526:0:17244:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.742533:0:17244:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00200000:0.0:1788201380.742544:0:17244:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788201380.742555:0:17244:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x407:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201380.742570:0:17244:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346370000 (ffff902348a2f368/0) at cl_lock_release():245 00000020:00010000:0.0:1788201380.742574:0:17244:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346370000 (ffff902348a2f368/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201380.742577:0:17244:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235306dfb0 (ffff902348a2f128/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201380.742580:0:17244:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346370000 (ffff902348a2f368/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201380.742583:0:17244:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235306dfb0 (ffff902348a2f368/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201380.742603:0:17244:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x407:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1788201380.743533:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201380.743545:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.743611:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902343731600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:3.0:1788201380.743617:0:5599:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.743626:0:5599:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.743635:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 3/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.743657:0:5599:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788201380.743817:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 2/0,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.743825:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1788201380.743901:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442718464:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201380.744013:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.744023:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731600/0x4605c7c97e1c532d lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d5166 expref: -99 pid: 17244 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.744041:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201380.744043:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201380.744045:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201380.746240:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442718464:192.168.201.130@tcp:103:ldlm_bl.0 00000080:00200000:1.0:1788201380.751615:0:17245:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201380.751644:0:17245:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201380.751662:0:17245:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:1.0:1788201380.751677:0:17245:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:1.0:1788201380.751684:0:17245:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201380.751691:0:17245:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201380.751694:0:17245:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:1.0:1788201380.751700:0:17245:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181cf3b58 00010000:00010000:1.0:1788201380.751704:0:17245:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:1.0:1788201380.751712:0:17245:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:1.0:1788201380.751729:0:17245:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201380.751737:0:17245:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=24 00000080:00200000:1.0:1788201380.751743:0:17245:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201380.751753:0:17245:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-6, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), flags=8 00000080:00200000:1.0:1788201380.751756:0:17245:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-6, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=getattr 00000002:00010000:1.0:1788201380.751772:0:17245:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-6,[0x0:0x0:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201380.751999:0:17245:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17245 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201380.752008:0:17245:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17245 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201380.752017:0:17245:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17245 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201380.752041:0:17245:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458bbb80 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17245:1875064442718592:192.168.201.130@tcp:101:checkstat.0 00000100:00100000:1.0:1788201380.752108:0:17245:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201380.760055:0:17245:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458bbb80 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17245:1875064442718592:192.168.201.130@tcp:101:checkstat.0 00010000:00010000:1.0:1788201380.760084:0:17245:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201380.760100:0:17245:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201380.760107:0:17245:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201380.760116:0:17245:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201380.760126:0:17245:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201380.760138:0:17245:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201380.760149:0:17245:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201380.760155:0:17245:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201380.760165:0:17245:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023458bbb80 x1875064442718592/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201396 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201380.760176:0:17245:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201380.760187:0:17245:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201380.760198:0:17245:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201380.760208:0:17245:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201380.760215:0:17245:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201380.760220:0:17245:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201380.760230:0:17245:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201380.760249:0:17245:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:1.0:1788201380.760255:0:17245:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788201380.760277:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201380.760286:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.760295:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345803000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201380.760300:0:13086:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.760310:0:13086:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.760319:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 3/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1788201380.760335:0:17245:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x408:0x0](ffff9023449ab248): rc = 0 00010000:00010000:2.0:1788201380.760337:0:13086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201380.760360:0:17245:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x408:0x0]: layout version change: 0 -> 0 00010000:00010000:2.0:1788201380.760376:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 2/0,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000080:00010000:1.0:1788201380.760382:0:17245:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x408:0x0](ffff9023449ab248) for lock 0x4605c7c97e1c52c4 00010000:00010000:2.0:1788201380.760387:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000080:00010000:1.0:1788201380.760402:0:17245:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x408:0x0](ffff9023449ab248) 00000080:00010000:1.0:1788201380.760407:0:17245:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52c4 from it ffffa0b181cf3b20 00010000:00010000:1.0:1788201380.760410:0:17245:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201380.760418:0:17245:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201380.760439:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.760449:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c533b lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d5197 expref: -99 pid: 17245 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000080:00200008:1.0:1788201380.760463:0:17245:0:(file.c:6283:ll_getattr_dentry()) START file file-6:[0x200000bd1:0x408:0x0](ffff9023449ab248), request_mask 2047, flags 0, foreign 0 00010000:00010000:2.0:1788201380.760469:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1788201380.760471:0:17245:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x408:0x0](ffff9023449ab248),name=file-6 00010000:00010000:2.0:1788201380.760473:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201380.760476:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201380.760484:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000002:00010000:1.0:1788201380.760490:0:17245:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x408:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201380.760499:0:17245:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000100:00100000:2.0:1788201380.760500:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442718720:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201380.760513:0:17245:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201380.760520:0:17245:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201380.760525:0:17245:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x408:0x0](ffff9023449ab248) 00000080:00010000:1.0:1788201380.760528:0:17245:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x408:0x0](ffff9023449ab248) for lock 0x4605c7c97e1c52c4 00000080:00010000:1.0:1788201380.760532:0:17245:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52c4 from it ffffa0b181cf3cb8 00010000:00010000:1.0:1788201380.760535:0:17245:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201380.760542:0:17245:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000080:00200000:1.0:1788201380.760566:0:17245:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x408:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788201380.760574:0:17245:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x408:0x0] 00000008:00010000:1.0:1788201380.760603:0:17245:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788201380.760607:0:17245:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023450459b0, osc lock ffff90235090c000, flags 0x400001002 00010000:00010000:1.0:1788201380.760621:0:17245:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201380.760632:0:17245:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:1.0:1788201380.760643:0:17245:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788201380.760648:0:17245:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201380.760655:0:17245:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000020:00200000:1.0:1788201380.760665:0:17245:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788201380.760676:0:17245:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x408:0x0] updating i_size 0 i_blocks 1 00000020:00010000:1.0:1788201380.760679:0:17245:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235844f090 (ffff9023500e7098/0) at cl_lock_release():245 00000020:00010000:1.0:1788201380.760682:0:17245:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235844f090 (ffff9023500e7098/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201380.760684:0:17245:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023450459b0 (ffff9023500e7878/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788201380.760688:0:17245:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235844f090 (ffff9023500e7098/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201380.760690:0:17245:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023450459b0 (ffff9023500e7098/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201380.760706:0:17245:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x408:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201380.766393:0:17246:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201380.766433:0:17246:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201380.766452:0:17246:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:3.0:1788201380.766466:0:17246:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:3.0:1788201380.766473:0:17246:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201380.766480:0:17246:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201380.766483:0:17246:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:3.0:1788201380.766490:0:17246:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181523b58 00010000:00010000:3.0:1788201380.766493:0:17246:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:3.0:1788201380.766501:0:17246:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:3.0:1788201380.766516:0:17246:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201380.766522:0:17246:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=24 00000080:00200000:3.0:1788201380.766528:0:17246:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201380.766536:0:17246:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=file-7, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), flags=8 00000080:00200000:3.0:1788201380.766541:0:17246:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=file-7, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988), intent=getattr 00000002:00010000:3.0:1788201380.766555:0:17246:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: file-7,[0x0:0x0:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201380.766596:0:17246:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17246 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.766604:0:17246:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17246 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.766614:0:17246:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17246 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201380.766641:0:17246:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17246:1875064442718848:192.168.201.130@tcp:101:checkstat.0 00000100:00100000:3.0:1788201380.766729:0:17246:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201380.773059:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442718720:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201380.775895:0:17246:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job checkstat:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17246:1875064442718848:192.168.201.130@tcp:101:checkstat.0 00010000:00010000:3.0:1788201380.775923:0:17246:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.775941:0:17246:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.775949:0:17246:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.775961:0:17246:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.775973:0:17246:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.775988:0:17246:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.775997:0:17246:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.776006:0:17246:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201380.776019:0:17246:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352688a80 x1875064442718848/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/640 e 0 to 0 dl 1788201396 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201380.776031:0:17246:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201380.776045:0:17246:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.776056:0:17246:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201380.776068:0:17246:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201380.776077:0:17246:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.776089:0:17246:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.776096:0:17246:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201380.776115:0:17246:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1788201380.776136:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201380.776145:0:5598:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.776154:0:5598:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351684c00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201380.776158:0:5598:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.776166:0:5598:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.776174:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 3/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.776190:0:5598:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.776226:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 2/0,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.776231:0:5598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788201380.776287:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442718976:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788201380.776682:0:5598:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.776691:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5342 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d51b3 expref: -99 pid: 17246 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.776706:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201380.776708:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201380.776710:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788201380.776856:0:17246:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201380.776965:0:17246:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x409:0x0](ffff9023449adb88): rc = 0 00000080:00200000:3.0:1788201380.776985:0:17246:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x409:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788201380.776996:0:17246:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x409:0x0](ffff9023449adb88) for lock 0x4605c7c97e1c52cb 00000080:00010000:3.0:1788201380.777010:0:17246:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x409:0x0](ffff9023449adb88) 00000080:00010000:3.0:1788201380.777014:0:17246:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52cb from it ffffa0b181523b20 00010000:00010000:3.0:1788201380.777018:0:17246:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201380.777027:0:17246:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:3.0:1788201380.777060:0:17246:0:(file.c:6283:ll_getattr_dentry()) START file file-7:[0x200000bd1:0x409:0x0](ffff9023449adb88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201380.777068:0:17246:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x409:0x0](ffff9023449adb88),name=file-7 00000002:00010000:3.0:1788201380.777085:0:17246:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x409:0x0]) in obj [0x200000bd1:0x402:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201380.777095:0:17246:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201380.777105:0:17246:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201380.777113:0:17246:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x402:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201380.777117:0:17246:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x409:0x0](ffff9023449adb88) 00000080:00010000:3.0:1788201380.777121:0:17246:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x409:0x0](ffff9023449adb88) for lock 0x4605c7c97e1c52cb 00000080:00010000:3.0:1788201380.777125:0:17246:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c52cb from it ffffa0b181523cb8 00010000:00010000:3.0:1788201380.777128:0:17246:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201380.777136:0:17246:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788201380.777159:0:17246:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x409:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1788201380.777167:0:17246:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x409:0x0] 00000008:00010000:3.0:1788201380.777192:0:17246:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788201380.777195:0:17246:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023483ef8b0, osc lock ffff902348866d10, flags 0x400001002 00010000:00010000:3.0:1788201380.777336:0:17246:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201380.777350:0:17246:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:3.0:1788201380.777363:0:17246:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788201380.777367:0:17246:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201380.777375:0:17246:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00200000:3.0:1788201380.777564:0:17246:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1788201380.777577:0:17246:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x409:0x0] updating i_size 0 i_blocks 1 00000020:00010000:3.0:1788201380.777582:0:17246:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235000c4a0 (ffff902346343d88/0) at cl_lock_release():245 00000020:00010000:3.0:1788201380.777585:0:17246:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235000c4a0 (ffff902346343d88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201380.777588:0:17246:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023483ef8b0 (ffff902346343758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788201380.777592:0:17246:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235000c4a0 (ffff902346343d88/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201380.777595:0:17246:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023483ef8b0 (ffff902346343d88/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201380.777619:0:17246:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x409:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:1.0:1788201380.780951:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442718976:192.168.201.130@tcp:103:ldlm_bl.0 00000080:00200000:2.0:1788201380.790241:0:17247:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201380.790289:0:17247:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201380.790317:0:17247:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:2.0:1788201380.790336:0:17247:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:2.0:1788201380.790345:0:17247:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201380.790354:0:17247:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201380.790358:0:17247:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:2.0:1788201380.790367:0:17247:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181673b58 00010000:00010000:2.0:1788201380.790372:0:17247:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:2.0:1788201380.790381:0:17247:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:2.0:1788201380.790672:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201380.790683:0:17247:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=8 00000080:00200008:2.0:1788201380.790703:0:17247:0:(file.c:6283:ll_getattr_dentry()) START file d102d.replay-single:[0x200000bd1:0x402:0x0](ffff9023449a1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201380.790710:0:17247:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988),name=d102d.replay-single 00000002:00010000:2.0:1788201380.790724:0:17247:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x402:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201380.790741:0:17247:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201380.790752:0:17247:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201380.790759:0:17247:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201380.790763:0:17247:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x402:0x0](ffff9023449a1988) 00000080:00010000:2.0:1788201380.790766:0:17247:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x402:0x0](ffff9023449a1988) for lock 0x4605c7c97e1c529a 00000080:00010000:2.0:1788201380.790770:0:17247:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c529a from it ffffa0b181673cb8 00010000:00010000:2.0:1788201380.790774:0:17247:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201380.790782:0:17247:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201380.790816:0:17247:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201380.790823:0:17247:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201380.790829:0:17247:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:2.0:1788201380.790837:0:17247:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:2.0:1788201380.790844:0:17247:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201380.790848:0:17247:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201380.790850:0:17247:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:2.0:1788201380.790854:0:17247:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181673bf8 00010000:00010000:2.0:1788201380.790857:0:17247:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:2.0:1788201380.790864:0:17247:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:2.0:1788201380.790873:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201380.790909:0:17247:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=266 00000080:00200000:2.0:1788201380.790914:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 44 00000080:00200008:2.0:1788201380.790922:0:17247:0:(file.c:1090:ll_file_open()) START file d102d.replay-single:[0x200000bd1:0x402:0x0](ffff9023449a1988), flags 704400 00000002:00010000:2.0:1788201380.790935:0:17247:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x402:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:2.0:1788201380.790983:0:17247:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107e00/0x4605c7c97e1c5349 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201380.790991:0:17247:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107e00/0x4605c7c97e1c5349 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201380.791003:0:17247:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201380.791016:0:17247:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234518c380 x1875064442719232/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201380.791026:0:17247:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107e00/0x4605c7c97e1c5349 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201380.791049:0:17247:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518c380 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442719232:192.168.201.130@tcp:101:rm.0 00000100:00100000:2.0:1788201380.791757:0:17247:0:(client.c:2702:ptlrpc_set_wait()) set ffff902352e83600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.794641:0:17247:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518c380 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442719232:192.168.201.130@tcp:101:rm.0 00010000:00010000:0.0:1788201380.794671:0:17247:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107e00/0x4605c7c97e1c5349 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.794688:0:17247:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107e00/0x4605c7c97e1c5349 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.794695:0:17247:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107e00/0x4605c7c97e1c5349 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.794703:0:17247:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107e00/0x4605c7c97e1c5349 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.794712:0:17247:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107e00/0x4605c7c97e1c5349 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.794720:0:17247:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345107e00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201380.794725:0:17247:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107e00/0x4605c7c97e1c5349 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.794733:0:17247:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107e00/0x4605c7c97e1c5349 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.794745:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107e00/0x4605c7c97e1c5349 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.794751:0:17247:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201380.794758:0:17247:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107e00/0x4605c7c97e1c5349 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.794764:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107e00/0x4605c7c97e1c5349 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201380.794784:0:17247:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff90234518c380 x1875064442719232/t17179869185(17179869185) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201396 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201380.794799:0:17247:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90234518c380 x1875064442719232/t17179869185(17179869185) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201396 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201380.794822:0:17247:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201380.794848:0:17247:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x402:0x0], updating i_size 7680 00000002:00100000:0.0:1788201380.794866:0:17247:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90234518c380 x1875064442719232/t17179869185(17179869185) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201396 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788201380.795012:0:17247:0:(file.c:6283:ll_getattr_dentry()) START file d102d.replay-single:[0x200000bd1:0x402:0x0](ffff9023449a1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201380.795020:0:17247:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988),name=d102d.replay-single 00000002:00010000:0.0:1788201380.795041:0:17247:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x402:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201380.795056:0:17247:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.795068:0:17247:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.795079:0:17247:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201380.795085:0:17247:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x402:0x0](ffff9023449a1988) 00000080:00010000:0.0:1788201380.795088:0:17247:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x402:0x0](ffff9023449a1988) for lock 0x4605c7c97e1c529a 00000080:00010000:0.0:1788201380.795094:0:17247:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c529a from it ffffa0b181673cd8 00010000:00010000:0.0:1788201380.795097:0:17247:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.795104:0:17247:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201380.795183:0:17247:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988) pos/size0/7680 32bit_api 0 00000002:00010000:0.0:1788201380.795195:0:17247:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x402:0x0]) in obj [0x200000bd1:0x402:0x0], intent: readdir flags 0 00010000:00010000:0.0:1788201380.795202:0:17247:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201380.795209:0:17247:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1788201380.796089:0:17247:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b2d80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442719360:192.168.201.130@tcp:37:rm.0 00000100:00100000:0.0:1788201380.796184:0:17247:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.806155:0:17247:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b2d80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442719360:192.168.201.130@tcp:37:rm.0 00010000:00010000:0.0:1788201380.807368:0:17247:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201380.807397:0:17247:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:0.0:1788201380.807519:0:17247:0:(file.c:479:ll_file_release()) START file d102d.replay-single:[0x200000bd1:0x402:0x0](ffff9023449a1988), flags 704000 00000002:00100000:0.0:1788201380.807583:0:17247:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff90234518c380 x1875064442719232/t17179869185(17179869185) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201396 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201380.807595:0:17247:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201380.807600:0:17247:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b2d80 x1875064442719616/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201380.807622:0:17247:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b2d80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442719616:192.168.201.130@tcp:35:rm.0 00000100:00100000:0.0:1788201380.807697:0:17247:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.814347:0:17247:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b2d80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442719616:192.168.201.130@tcp:35:rm.0 00000002:00100000:0.0:1788201380.814380:0:17247:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff90234518c380 x1875064442719232/t17179869185(17179869185) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201396 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201380.814471:0:17247:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201380.814490:0:17247:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201380.814508:0:17247:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:0.0:1788201380.814521:0:17247:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:0.0:1788201380.814529:0:17247:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201380.814535:0:17247:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201380.814538:0:17247:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:0.0:1788201380.814545:0:17247:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181673bf8 00010000:00010000:0.0:1788201380.814549:0:17247:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:0.0:1788201380.814558:0:17247:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:0.0:1788201380.814573:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201380.814580:0:17247:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d102d.replay-single, flags=266 00000080:00200000:0.0:1788201380.814587:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 44 00000080:00200008:0.0:1788201380.814594:0:17247:0:(file.c:1090:ll_file_open()) START file d102d.replay-single:[0x200000bd1:0x402:0x0](ffff9023449a1988), flags 704400 00000002:00010000:0.0:1788201380.814610:0:17247:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x402:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:0.0:1788201380.814654:0:17247:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5350 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.814661:0:17247:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5350 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201380.814673:0:17247:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201380.814679:0:17247:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b2d80 x1875064442719872/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201380.814686:0:17247:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5350 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201380.814703:0:17247:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b2d80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442719872:192.168.201.130@tcp:101:rm.0 00000100:00100000:0.0:1788201380.815625:0:17247:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.821859:0:17247:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b2d80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442719872:192.168.201.130@tcp:101:rm.0 00010000:00010000:0.0:1788201380.821920:0:17247:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5350 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d51cf expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.821937:0:17247:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5350 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d51cf expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.821948:0:17247:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5350 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d51cf expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.821959:0:17247:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5350 lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d51cf expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201380.821976:0:17247:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff9023488b2d80 x1875064442719872/t17179869187(17179869187) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201396 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201380.821990:0:17247:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b2d80 x1875064442719872/t17179869187(17179869187) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201396 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201380.822006:0:17247:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5350 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d51cf expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.822017:0:17247:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1026 (0 0) 00000080:00200000:0.0:1788201380.822024:0:17247:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201380.822045:0:17247:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x402:0x0], updating i_size 7680 00000080:00010000:0.0:1788201380.822050:0:17247:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x402:0x0](ffff9023449a1988) for lock 0x4605c7c97e1c5350 00000080:00010000:0.0:1788201380.822057:0:17247:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5350 from it ffffa0b181673bd0 00010000:00010000:0.0:1788201380.822061:0:17247:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5350 lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d51cf expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.822070:0:17247:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5350 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d51cf expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1788201380.822098:0:17247:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b2d80 x1875064442719872/t17179869187(17179869187) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201396 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788201380.822176:0:17247:0:(file.c:6283:ll_getattr_dentry()) START file d102d.replay-single:[0x200000bd1:0x402:0x0](ffff9023449a1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201380.822183:0:17247:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988),name=d102d.replay-single 00000002:00010000:0.0:1788201380.822202:0:17247:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x402:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201380.822212:0:17247:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201380.822223:0:17247:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201380.822229:0:17247:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201380.822233:0:17247:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x402:0x0](ffff9023449a1988) 00000080:00010000:0.0:1788201380.822236:0:17247:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x402:0x0](ffff9023449a1988) for lock 0x4605c7c97e1c529a 00000080:00010000:0.0:1788201380.822240:0:17247:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c529a from it ffffa0b181673cd8 00010000:00010000:0.0:1788201380.822242:0:17247:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201380.822249:0:17247:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:0.0:1788201380.822389:0:17247:0:(llite_lib.c:2912:ll_statfs()) VFS Op:sb=lustre (ffff902350a11000) d102d.replay-single proj=0 00000080:00000004:0.0:1788201380.822394:0:17247:0:(obd_class.h:1091:obd_statfs()) lustre-clilmv-ffff902350a11000: age 593, max_age 656 00800000:00000004:0.0:1788201380.822400:0:17247:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: age 657, max_age 656 00800000:00000004:0.0:1788201380.822402:0:17247:0:(obd_class.h:1131:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: use ffff902350be5190 cache blocks 17239/17271 objects 531669/532010 00000080:00000004:0.0:1788201380.822407:0:17247:0:(obd_class.h:1117:obd_statfs()) lustre-clilmv-ffff902350a11000: update ffff902350be0450 cache blocks 17239/17271 objects 531669/532010 00000080:00000004:0.0:1788201380.822411:0:17247:0:(llite_lib.c:2764:ll_statfs_internal()) MDC blocks 17239/17271 objects 531669/532010 00000080:00000004:0.0:1788201380.822413:0:17247:0:(obd_class.h:1091:obd_statfs()) lustre-clilov-ffff902350a11000: age 593, max_age 656 00020000:00000004:0.0:1788201380.822438:0:17247:0:(obd_class.h:1057:obd_statfs_async()) lustre-OST0000-osc-ffff902350a11000: age 657, max_age 656 00000008:00000004:0.0:1788201380.822442:0:17247:0:(osc_request.c:3416:osc_statfs_async()) lustre-OST0000-osc-ffff902350a11000: use ffff902350be1e10 cache blocks 3667/3683 objects 117728/118668 00020000:00000004:0.0:1788201380.822447:0:17247:0:(obd_class.h:1057:obd_statfs_async()) lustre-OST0001-osc-ffff902350a11000: age 657, max_age 656 00000008:00000004:0.0:1788201380.822449:0:17247:0:(osc_request.c:3416:osc_statfs_async()) lustre-OST0001-osc-ffff902350a11000: use ffff902351d65190 cache blocks 3666/3683 objects 117792/118189 00000080:00000004:0.0:1788201380.822456:0:17247:0:(obd_class.h:1117:obd_statfs()) lustre-clilov-ffff902350a11000: update ffff902350be37d0 cache blocks 7333/7366 objects 235520/236857 00000080:00000004:0.0:1788201380.822459:0:17247:0:(llite_lib.c:2775:ll_statfs_internal()) OSC blocks 7333/7366 objects 235520/236857 00000080:00200000:0.0:1788201380.822473:0:17247:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988) pos/size0/7680 32bit_api 0 00000080:00200000:0.0:1788201380.822510:0:17247:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988) pos/size18446744073709551614/7680 32bit_api 0 00000080:00200000:0.0:1788201380.822585:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201380.822596:0:17247:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-3, flags=0 00000080:00200000:0.0:1788201380.822607:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201380.822611:0:17247:0:(namei.c:2418:ll_unlink()) VFS Op:name=file-3, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988) 00010000:00010000:0.0:1788201380.822675:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788201380.822689:0:17247:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5350 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d51cf expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.822936:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.822959:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x405:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.822974:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5293 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d4ef0 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.822985:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x405:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.823015:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.823022:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5293 lrc: 1/0,0 mode: --/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4ef0 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.823028:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788201380.823032:0:17247:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1788201380.823035:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c52af lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x405:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5040 expref: -99 pid: 17235 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.823045:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0000/0x4605c7c97e1c5293 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x405:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4ef0 expref: -99 pid: 16413 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.823058:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106400/0x4605c7c97e1c529a lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d4fec expref: -99 pid: 17229 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000020:00100000:0.0:1788201380.823072:0:17247:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201380.823080:0:17247:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b1500 x1875064442720128/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201380.823097:0:17247:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1500 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442720128:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201380.823169:0:17247:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.831921:0:17247:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1500 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442720128:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201380.832007:0:17247:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x405:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201380.832020:0:17247:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201380.832075:0:17247:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x405:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201380.832088:0:17247:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x405:0x0](ffff9023449b5348) 00000008:00010000:0.0:1788201380.832125:0:17247:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 1788201325 1788201325 1788201325 by oinfo size 0 blocks 1 [cma]time 1788201326 1788201325 1788201325 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x443:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.832195:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201380.832208:0:17247:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-4, flags=0 00000080:00200000:0.0:1788201380.832213:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201380.832218:0:17247:0:(namei.c:2418:ll_unlink()) VFS Op:name=file-4, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988) 00010000:00010000:0.0:1788201380.832263:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d505c expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.832287:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x406:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.832311:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.832327:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x406:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.832339:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c528c lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d4ee9 expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.832354:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x406:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.832389:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d505c expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.832398:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.832404:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c528c lrc: 1/0,0 mode: --/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4ee9 expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.832417:0:17247:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1788201380.832430:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1c00/0x4605c7c97e1c52b6 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d505c expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.832442:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c52ee lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x406:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d50e1 expref: -99 pid: 17240 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.832452:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1400/0x4605c7c97e1c528c lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x406:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4ee9 expref: -99 pid: 16414 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201380.832473:0:17247:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201380.832482:0:17247:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b2a00 x1875064442720384/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201380.832504:0:17247:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b2a00 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442720384:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201380.832604:0:17247:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.839691:0:17247:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b2a00 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442720384:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201380.839763:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201380.839774:0:17247:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-7, flags=0 00000080:00200000:0.0:1788201380.839782:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201380.839788:0:17247:0:(namei.c:2418:ll_unlink()) VFS Op:name=file-7, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988) 00010000:00010000:0.0:1788201380.839985:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5135 expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.840014:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x409:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.840039:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201380.840057:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x409:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.840069:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c5277 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d4ed4 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.840084:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x409:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.840122:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5135 expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.840129:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201380.840135:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c5277 lrc: 1/0,0 mode: --/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4ed4 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.840140:0:17247:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1788201380.840143:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5303 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5135 expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.840153:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0800/0x4605c7c97e1c52cb lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x409:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d50b0 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201380.840161:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0e00/0x4605c7c97e1c5277 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x409:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4ed4 expref: -99 pid: 16417 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201380.840179:0:17247:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201380.840188:0:17247:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b0700 x1875064442720640/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201380.840214:0:17247:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b0700 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442720640:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201380.840276:0:17247:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.846853:0:17247:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b0700 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442720640:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201380.846955:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201380.846969:0:17247:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-5, flags=0 00000080:00200000:0.0:1788201380.846978:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201380.846984:0:17247:0:(namei.c:2418:ll_unlink()) VFS Op:name=file-5, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988) 00010000:00010000:0.0:1788201380.847069:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d50fd expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.847101:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x407:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.847126:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00000080:00200000:0.0:1788201380.847144:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x407:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.847157:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106e00/0x4605c7c97e1c5285 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d4ee2 expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.847171:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x407:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.847207:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d50fd expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.847215:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788201380.847221:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106e00/0x4605c7c97e1c5285 lrc: 1/0,0 mode: --/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4ee2 expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.847231:0:17247:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1788201380.847234:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c52f5 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d50fd expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.847245:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0a00/0x4605c7c97e1c52bd lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x407:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5078 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788201380.847254:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106e00/0x4605c7c97e1c5285 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x407:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4ee2 expref: -99 pid: 16415 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201380.847276:0:17247:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201380.847290:0:17247:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b2300 x1875064442720896/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201380.847317:0:17247:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b2300 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442720896:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201380.847379:0:17247:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.852006:0:17247:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b2300 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442720896:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201380.852075:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201380.852086:0:17247:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-2, flags=0 00000080:00200000:0.0:1788201380.852094:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201380.852099:0:17247:0:(namei.c:2418:ll_unlink()) VFS Op:name=file-2, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988) 00010000:00010000:0.0:1788201380.852183:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.852212:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x404:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.852238:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5269 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d4f28 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.852256:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x404:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.852294:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.852302:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5269 lrc: 1/0,0 mode: --/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4f28 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.852308:0:17247:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201380.852310:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1e00/0x4605c7c97e1c52a8 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x404:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5024 expref: -99 pid: 17232 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.852321:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5269 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x404:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4f28 expref: -99 pid: 16412 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201380.852342:0:17247:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201380.852351:0:17247:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b0e00 x1875064442721152/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201380.852371:0:17247:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b0e00 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442721152:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201380.852468:0:17247:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.855862:0:17247:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b0e00 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442721152:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201380.856039:0:17247:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x404:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201380.856051:0:17247:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201380.856104:0:17247:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x404:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201380.856118:0:17247:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x404:0x0](ffff9023449a7448) 00000008:00010000:0.0:1788201380.856157:0:17247:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 1788201325 1788201325 1788201325 by oinfo size 0 blocks 1 [cma]time 1788201326 1788201325 1788201325 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x442:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.856230:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201380.856240:0:17247:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-6, flags=0 00000080:00200000:0.0:1788201380.856248:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201380.856252:0:17247:0:(namei.c:2418:ll_unlink()) VFS Op:name=file-6, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988) 00010000:00010000:0.0:1788201380.856300:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5119 expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.856327:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x408:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.856351:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000080:00200000:0.0:1788201380.856367:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x408:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.856377:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1000/0x4605c7c97e1c5270 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d4ecd expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.856390:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x408:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.856439:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5119 expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.856447:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788201380.856454:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1000/0x4605c7c97e1c5270 lrc: 1/0,0 mode: --/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4ecd expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.856459:0:17247:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1788201380.856462:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c52fc lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5119 expref: -99 pid: 17241 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.856474:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1800/0x4605c7c97e1c52c4 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x408:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5094 expref: -99 pid: 17238 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788201380.856482:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1000/0x4605c7c97e1c5270 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x408:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4ecd expref: -99 pid: 16416 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201380.856501:0:17247:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201380.856511:0:17247:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b3800 x1875064442721408/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201380.856532:0:17247:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b3800 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442721408:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201380.856638:0:17247:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa3c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.861507:0:17247:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b3800 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442721408:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201380.861588:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201380.861599:0:17247:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=file-1, flags=0 00000080:00200000:0.0:1788201380.861610:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201380.861615:0:17247:0:(namei.c:2418:ll_unlink()) VFS Op:name=file-1, dir=[0x200000bd1:0x402:0x0](ffff9023449a1988) 00010000:00010000:0.0:1788201380.861705:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.861732:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x403:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.861758:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c527e lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d4edb expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201380.861775:0:17247:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x403:0x0]: losing layout lock 00010000:00010000:0.0:1788201380.861813:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.861822:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c527e lrc: 1/0,0 mode: --/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4edb expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.861828:0:17247:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201380.861830:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db1200/0x4605c7c97e1c52a1 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x403:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5008 expref: -99 pid: 17229 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201380.861843:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342db0200/0x4605c7c97e1c527e lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x403:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d4edb expref: -99 pid: 16411 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201380.861866:0:17247:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201380.861913:0:17247:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b1c00 x1875064442721664/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201380.861946:0:17247:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1c00 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442721664:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201380.862006:0:17247:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa3c0 going to sleep for 11 seconds 00000080:00200000:3.0:1788201380.864994:0:17238:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x406:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201380.865010:0:17238:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201380.865057:0:17238:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x406:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201380.865069:0:17238:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x406:0x0](ffff902344979148) 00000008:00010000:3.0:1788201380.865108:0:17238:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 1788201325 1788201325 1788201325 by oinfo size 0 blocks 1 [cma]time 1788201326 1788201325 1788201325 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x443:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1788201380.869086:0:17247:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1c00 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442721664:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201380.869173:0:17247:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x403:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201380.869185:0:17247:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201380.869236:0:17247:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x403:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201380.869251:0:17247:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x403:0x0](ffff90234497ec08) 00000008:00010000:0.0:1788201380.869290:0:17247:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 1788201325 1788201325 1788201325 by oinfo size 0 blocks 1 [cma]time 1788201326 1788201325 1788201325 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x442:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200008:0.0:1788201380.869357:0:17247:0:(file.c:479:ll_file_release()) START file d102d.replay-single:[0x200000bd1:0x402:0x0](ffff9023449a1988), flags 704000 00010000:00010000:0.0:1788201380.869379:0:17247:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5350 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d51cf expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201380.869412:0:17247:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201380.869437:0:17247:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201380.869447:0:17247:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:0.0:1788201380.869456:0:17247:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:0.0:1788201380.869463:0:17247:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201380.869470:0:17247:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201380.869473:0:17247:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:0.0:1788201380.869479:0:17247:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181673c10 00010000:00010000:0.0:1788201380.869484:0:17247:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:0.0:1788201380.869492:0:17247:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200000:0.0:1788201380.869507:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201380.869524:0:17247:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d102d.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=0 00000080:00200000:0.0:1788201380.869529:0:17247:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d102d.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=getattr 00000002:00010000:0.0:1788201380.869540:0:17247:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d102d.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201380.869583:0:17247:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c5357 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.869590:0:17247:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c5357 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.869599:0:17247:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c5357 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17247 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201380.869618:0:17247:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1f80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442721792:192.168.201.130@tcp:101:rm.0 00000100:00100000:0.0:1788201380.869681:0:17247:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa3c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.874173:0:17247:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1f80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442721792:192.168.201.130@tcp:101:rm.0 00010000:00010000:0.0:1788201380.874201:0:17247:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c5357 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52b6 expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.874219:0:17247:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c5357 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52b6 expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.874227:0:17247:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c5357 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52b6 expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.874238:0:17247:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c5357 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d52b6 expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.874250:0:17247:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c5357 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52b6 expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.874263:0:17247:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c5357 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d52b6 expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.874272:0:17247:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c5357 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d52b6 expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.874285:0:17247:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c5357 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d52b6 expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201380.874300:0:17247:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023488b1f80 x1875064442721792/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201396 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201380.874329:0:17247:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c5357 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52b6 expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.874339:0:17247:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1026 (0 0) 00000080:00200000:0.0:1788201380.874346:0:17247:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201380.874370:0:17247:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x402:0x0], updating i_size 7680 00000080:00200000:0.0:1788201380.874373:0:17247:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x402:0x0](ffff9023449a1988): rc = 0 00000080:00010000:0.0:1788201380.874379:0:17247:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x402:0x0](ffff9023449a1988) for lock 0x4605c7c97e1c5357 00000080:00010000:0.0:1788201380.874392:0:17247:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x402:0x0](ffff9023449a1988) 00000080:00010000:0.0:1788201380.874396:0:17247:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5357 from it ffffa0b181673e00 00010000:00010000:0.0:1788201380.874400:0:17247:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c5357 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d52b6 expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201380.874409:0:17247:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c5357 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52b6 expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201380.874462:0:17247:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201380.874481:0:17247:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201380.874491:0:17247:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:0.0:1788201380.874506:0:17247:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200000:0.0:1788201380.874511:0:17247:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201380.874515:0:17247:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201380.874518:0:17247:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5078 00000080:00010000:0.0:1788201380.874533:0:17247:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5078 from it ffffa0b181673d88 00010000:00010000:0.0:1788201380.874537:0:17247:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:0.0:1788201380.874544:0:17247:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00200000:0.0:1788201380.874553:0:17247:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 3 00000080:00200000:0.0:1788201380.874560:0:17247:0:(namei.c:2328:ll_rmdir()) VFS Op:name=d102d.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908) 00010000:00010000:0.0:1788201380.874581:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:0.0:1788201380.874601:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c5357 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d52b6 expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.874611:0:17247:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5350 lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d51cf expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00100000:0.0:1788201380.874652:0:17247:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b2d80 x1875064442719872/t17179869187(17179869187) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201396 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201380.874667:0:17247:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201380.874672:0:17247:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b1f80 x1875064442722048/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201380.874691:0:17247:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1f80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442722048:192.168.201.130@tcp:35:rm.0 00000100:00100000:0.0:1788201380.875010:0:17247:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa3c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.878512:0:17247:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1f80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442722048:192.168.201.130@tcp:35:rm.0 00000002:00100000:0.0:1788201380.878622:0:17247:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b2d80 x1875064442719872/t17179869187(17179869187) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201396 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788201380.878696:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c5357 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d52b6 expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.878705:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5350 lrc: 1/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d51cf expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201380.878713:0:17247:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:0.0:1788201380.878717:0:17247:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1788201380.878720:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c5357 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x402:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d52b6 expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201380.878732:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5350 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d51cf expref: -99 pid: 17247 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201380.878744:0:17247:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c5078 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d4dae expref: -99 pid: 16390 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000020:00100000:0.0:1788201380.878767:0:17247:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201380.878774:0:17247:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b1f80 x1875064442722304/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201380.878794:0:17247:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1f80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442722304:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201380.879116:0:17247:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa3c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201380.884735:0:17247:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1f80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17247:1875064442722304:192.168.201.130@tcp:36:rm.0 00000100:00100000:3.0:1788201381.177771:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201381.177799:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201381.177822:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458262912 00000100:00100000:3.0:1788201381.177831:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458262912 00000100:00100000:3.0:1788201381.177869:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458263040 00000100:00100000:3.0:1788201381.177871:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458263040 00000100:00100000:3.0:1788201381.177903:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1333 00000100:00100000:3.0:1788201381.177912:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902352688a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13443:x1875064458262912:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201381.177930:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201381.177996:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201381.178002:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902352688a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13443:x1875064458262912:12345-192.168.201.130@tcp:104: Request processed in 90us (242us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201381.178008:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1333 00000100:00100000:3.0:1788201381.178022:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1334 00000100:00100000:3.0:1788201381.178027:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90235268b100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13450:x1875064458263040:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201381.178035:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201381.178059:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1788201381.178064:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90235268b100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13450:x1875064458263040:12345-192.168.201.130@tcp:104: Request processed in 36us (264us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201381.178069:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1334 00010000:00010000:3.0:1788201381.178628:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1788201381.178635:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201381.178646:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839ce00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201381.178651:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201381.178733:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201381.178742:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201381.178764:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201381.178771:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5135 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d4a2e expref: -99 pid: 16413 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201381.178789:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201381.178793:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201381.178799:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351684a00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201381.178803:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201381.178836:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 2/0,0 mode: --/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201381.178843:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201381.178848:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 1/0,0 mode: --/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201381.178856:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000391fa793/0x4605c7c97e1c512e lrc: 0/0,0 mode: --/PR res: [0x240000400:0x443:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d4a27 expref: -99 pid: 16414 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201381.178871:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201381.178873:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201381.178907:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201381.178913:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201381.178929:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442722432:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201381.178991:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201381.178998:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442722560:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201381.185225:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442722560:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201381.185243:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442722432:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201381.509490:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201382.532186:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201383.553179:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201384.577793:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201385.602034:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201385.857168:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442722688:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201385.857279:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442722944:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201385.857315:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788201385.857321:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442722816:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201385.857357:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442723072:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201385.859631:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442722688:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201385.860696:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442723072:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201385.860711:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442722944:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201385.864191:0:2361:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 17179869196 gen 2 00000100:00100000:2.0:1788201385.864239:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442722816:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201386.215399:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201386.215438:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201386.215472:0:5590:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1788201386.215490:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458264704 00000100:00100000:3.0:1788201386.215499:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458264704 00000100:00100000:3.0:1788201386.215564:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458264832 00000100:00100000:3.0:1788201386.215568:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458264832 00000100:00100000:3.0:1788201386.215595:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1335 00000100:00100000:3.0:1788201386.215605:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902352689180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+5:12363:x1875064458264704:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201386.215624:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000010000 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201386.215706:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201386.215713:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902352689180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+5:12363:x1875064458264704:12345-192.168.201.130@tcp:104: Request processed in 110us (323us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201386.215720:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1335 00000100:00100000:3.0:1788201386.215737:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1336 00000100:00100000:3.0:1788201386.215744:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023584a6450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13444:x1875064458264832:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201386.215752:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201386.215783:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1788201386.215789:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023584a6450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13444:x1875064458264832:12345-192.168.201.130@tcp:104: Request processed in 46us (355us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201386.215795:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1336 00010000:00010000:3.0:1788201386.215926:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1788201386.215933:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000010000 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201386.215942:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351684600 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201386.215948:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400010000 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201386.216023:0:13084:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x409:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788201386.216035:0:13084:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x409:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1788201386.216096:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 2/0,0 mode: --/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400010020 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201386.216104:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201386.216126:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 1/0,0 mode: --/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400010020 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201386.216135:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d41f4c9e/0x4605c7c97e1c513c lrc: 0/0,0 mode: --/PR res: [0x280000400:0x445:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400010020 nid: local remote: 0x9383f985bc4d4a35 expref: -99 pid: 16417 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201386.216155:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201386.216159:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201386.216166:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345802e00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201386.216170:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201386.216202:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 2/0,0 mode: --/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201386.216209:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201386.216216:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 1/0,0 mode: --/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201386.216223:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000a2e9ff36/0x4605c7c97e1c514a lrc: 0/0,0 mode: --/PR res: [0x280000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d4a5f expref: -99 pid: 16411 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201386.216237:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201386.216239:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201386.216241:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201386.216256:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442723200:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201386.216308:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201386.216314:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442723328:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788201386.218477:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201386.229661:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201386.229741:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458264576 00000100:00100000:2.0:1788201386.229751:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458264576 00000100:00100000:2.0:1788201386.229848:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1337 00000100:00100000:2.0:1788201386.229860:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902352689180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13446:x1875064458264576:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201386.230151:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c516d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d4a6d expref: -99 pid: 16421 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201386.230230:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201386.230238:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902352689180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13446:x1875064458264576:12345-192.168.201.130@tcp:104: Request processed in 380us (588us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201386.230245:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1337 00010000:00010000:0.0:1788201386.233922:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201386.233938:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c516d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d4a6d expref: -99 pid: 16421 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201386.233953:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839d200 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:0.0:1788201386.233961:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c516d lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d4a6d expref: -99 pid: 16421 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201386.234056:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c516d lrc: 2/0,0 mode: --/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d4a6d expref: -99 pid: 16421 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201386.234063:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201386.234103:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c516d lrc: 1/0,0 mode: --/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d4a6d expref: -99 pid: 16421 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201386.234112:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000087d291d1/0x4605c7c97e1c516d lrc: 0/0,0 mode: --/PR res: [0x280000400:0x444:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d4a6d expref: -99 pid: 16421 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201386.234127:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201386.234129:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201386.234131:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201386.234150:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442723456:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201386.239858:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201386.240311:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458264960 00000100:00100000:3.0:1788201386.240321:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458264960 00000100:00100000:3.0:1788201386.240406:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1338 00000100:00100000:3.0:1788201386.240416:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90235268b480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13446:x1875064458264960:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201386.241069:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000010000 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788201386.241196:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201386.241204:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90235268b480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13446:x1875064458264960:12345-192.168.201.130@tcp:104: Request processed in 789us (1355us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201386.241218:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1338 00000100:00100000:3.0:1788201386.241286:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442723328:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201386.241301:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442723200:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201386.241698:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201386.241706:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000010000 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201386.241777:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351685800 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:1.0:1788201386.241784:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400010000 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1788201386.241841:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x407:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201386.241853:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x407:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201386.241936:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400010020 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201386.241943:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201386.241966:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400010020 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201386.241974:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000009e287e23/0x4605c7c97e1c5197 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400010020 nid: local remote: 0x9383f985bc4d4a66 expref: -99 pid: 16415 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201386.241994:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201386.241996:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201386.241999:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201386.242074:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442723584:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201386.244386:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442723456:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201386.251067:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442723584:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201386.626773:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201387.128225:0:17804:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 103: Check otr_next_id overflow ==== 14:36:25 (1788201385) 00010000:00010000:0.0:1788201387.649195:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788201388.619808:0:17856:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201388.619846:0:17856:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201388.620609:0:17856:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 17856 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.620628:0:17856:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 17856 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.620641:0:17856:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 17856 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201388.620677:0:17856:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job mkdir:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17856:1875064442723712:192.168.201.130@tcp:101:mkdir.0 00000100:00100000:3.0:1788201388.620862:0:17856:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448f00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201388.627761:0:17856:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job mkdir:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17856:1875064442723712:192.168.201.130@tcp:101:mkdir.0 00010000:00010000:3.0:1788201388.627791:0:17856:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.627804:0:17856:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.627817:0:17856:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.627828:0:17856:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.627846:0:17856:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.627854:0:17856:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.627861:0:17856:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201388.627871:0:17856:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352688a80 x1875064442723712/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201404 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201388.627931:0:17856:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.627943:0:17856:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201388.627949:0:17856:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201388.627978:0:17856:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:3.0:1788201388.627984:0:17856:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c535e 00000080:00010000:3.0:1788201388.627989:0:17856:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201388.627992:0:17856:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c535e from it ffffa0b181be3e18 00010000:00010000:3.0:1788201388.627997:0:17856:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.628005:0:17856:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201388.628033:0:17856:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 101 00000080:00200000:3.0:1788201388.628064:0:17856:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201388.628076:0:17856:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201388.628085:0:17856:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201388.628094:0:17856:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201388.628101:0:17856:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201388.628104:0:17856:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201388.628107:0:17856:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c535e 00000080:00010000:3.0:1788201388.628110:0:17856:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c535e from it ffffa0b181be3be0 00010000:00010000:3.0:1788201388.628113:0:17856:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201388.628120:0:17856:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201388.628127:0:17856:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201388.628140:0:17856:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d103.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=1536 00000080:00200000:3.0:1788201388.628143:0:17856:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201388.628148:0:17856:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201388.628154:0:17856:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201388.628228:0:17856:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201388.628235:0:17856:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201388.628239:0:17856:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201388.628242:0:17856:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c535e 00000080:00010000:3.0:1788201388.628245:0:17856:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c535e from it ffffa0b181be3ce0 00010000:00010000:3.0:1788201388.628249:0:17856:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201388.628263:0:17856:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201388.628272:0:17856:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 3 00000080:00200000:3.0:1788201388.628277:0:17856:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201388.628282:0:17856:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201388.628298:0:17856:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201388.628434:0:17856:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201388.628442:0:17856:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201388.628446:0:17856:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201388.628448:0:17856:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c535e 00000080:00010000:3.0:1788201388.628453:0:17856:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c535e from it ffffa0b181be3dc8 00010000:00010000:3.0:1788201388.628456:0:17856:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201388.628463:0:17856:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788201388.628473:0:17856:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 3 00000080:00200000:3.0:1788201388.628478:0:17856:0:(namei.c:2239:do_mkdir()) VFS Op:name=d103.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908) 00010000:00010000:3.0:1788201388.628511:0:17856:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201388.628553:0:17856:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201388.628560:0:17856:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201388.628562:0:17856:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c535e lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d52fc expref: -99 pid: 17856 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000020:00100000:3.0:1788201388.628585:0:17856:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201388.628592:0:17856:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352688a80 x1875064442723968/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 568/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201388.628610:0:17856:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job mkdir:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17856:1875064442723968:192.168.201.130@tcp:36:mkdir.0 00000100:00100000:3.0:1788201388.628687:0:17856:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448f00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201388.643591:0:17856:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job mkdir:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17856:1875064442723968:192.168.201.130@tcp:36:mkdir.0 00000080:00200000:3.0:1788201388.643642:0:17856:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148) 00000080:00200000:3.0:1788201388.643650:0:17856:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x40a:0x0], updating i_size 512 00010000:00010000:0.0:1788201388.673438:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788201388.729457:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201388.729488:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201388.729554:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.729564:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.729575:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201388.729600:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442724096:192.168.201.130@tcp:101:createmany.0 00000100:00100000:3.0:1788201388.729783:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448f00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201388.741178:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442724096:192.168.201.130@tcp:101:createmany.0 00010000:00010000:3.0:1788201388.741212:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.741227:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.741238:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.741249:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.741263:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.741272:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.741280:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201388.741301:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff90235268b100 x1875064442724096/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201404 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201388.741315:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.741330:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201388.741337:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201388.741366:0:17857:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:3.0:1788201388.741373:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:3.0:1788201388.741380:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201388.741383:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:3.0:1788201388.741395:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.741405:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201388.741431:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201388.741452:0:17857:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d103.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=25 00000080:00200000:3.0:1788201388.741456:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d103.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=0 00000002:00010000:3.0:1788201388.741475:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d103.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1788201388.741514:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c536c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.741522:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c536c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.741537:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c536c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201388.741558:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442724224:192.168.201.130@tcp:101:createmany.0 00000100:00100000:3.0:1788201388.741644:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448f00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201388.747106:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442724224:192.168.201.130@tcp:101:createmany.0 00010000:00010000:3.0:1788201388.747130:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c536c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5334 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.747147:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c536c lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5334 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.747153:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c536c lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5334 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.747161:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c536c lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5334 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.747169:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c536c lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5334 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.747181:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c536c lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5334 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.747193:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c536c lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5334 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.747200:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c536c lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5334 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201388.747212:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff90235268b100 x1875064442724224/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201404 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201388.747224:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c536c lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5334 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.747234:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1034 (0 0) 00000080:00200000:3.0:1788201388.747239:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201388.747259:0:17857:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x40a:0x0], updating i_size 512 00000080:00200000:3.0:1788201388.747261:0:17857:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x40a:0x0](ffff902344979148): rc = 0 00000080:00010000:3.0:1788201388.747266:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x40a:0x0](ffff902344979148) for lock 0x4605c7c97e1c536c 00000080:00010000:3.0:1788201388.747275:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x40a:0x0](ffff902344979148) 00000080:00010000:3.0:1788201388.747278:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c536c from it ffffa0b181c53ac8 00010000:00010000:3.0:1788201388.747280:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c536c lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5334 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.747297:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c536c lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5334 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201388.747328:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:3.0:1788201388.747350:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:3.0:1788201388.747356:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-0, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff902343d62000, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788201388.747364:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-0, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:3.0:1788201388.747402:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-0,[0x200000bd1:0x40b:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201388.747446:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.747451:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201388.747460:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201388.747466:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90235268b100 x1875064442724480/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201388.747475:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201388.747493:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442724480:192.168.201.130@tcp:101:createmany.0 00000100:00100000:3.0:1788201388.747629:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448f00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201388.768086:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442724480:192.168.201.130@tcp:101:createmany.0 00010000:00010000:3.0:1788201388.768116:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.768142:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.768151:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.768163:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.768175:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.768188:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.768196:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.768203:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201388.768214:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff90235268b100 x1875064442724480/t17179869198(17179869198) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201388.768229:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90235268b100 x1875064442724480/t17179869198(17179869198) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201388.768243:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201388.768257:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.768268:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1035 (0 0) 00000080:00200000:3.0:1788201388.768274:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201388.768298:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-0, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:3.0:1788201388.768329:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x40b:0x0](ffff90234497f448) 00000080:00200000:3.0:1788201388.769157:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x40b:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201388.769170:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x40b:0x0](ffff90234497f448) 00000080:00010000:3.0:1788201388.769174:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x40b:0x0](ffff90234497f448) for lock 0x4605c7c97e1c5373 00000080:00010000:3.0:1788201388.769185:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x40b:0x0](ffff90234497f448) 00000080:00200008:3.0:1788201388.769198:0:17857:0:(file.c:1090:ll_file_open()) START file t-0:[0x200000bd1:0x40b:0x0](ffff90234497f448), flags 100102 00000002:00100000:3.0:1788201388.769221:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90235268b100 x1875064442724480/t17179869198(17179869198) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201388.769261:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5373 from it ffff9023533eade0 00010000:00010000:3.0:1788201388.769267:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.769276:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201388.769309:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40b:0x0](ffff90234497f448), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201388.769339:0:17857:0:(file.c:479:ll_file_release()) START file t-0:[0x200000bd1:0x40b:0x0](ffff90234497f448), flags 100002 00000002:00100000:3.0:1788201388.769383:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff90235268b100 x1875064442724480/t17179869198(17179869198) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201388.770009:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201388.770017:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352688e00 x1875064442724736/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201388.770043:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442724736:192.168.201.130@tcp:35:createmany.0 00000100:00100000:3.0:1788201388.770198:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584489c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201388.783004:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442724736:192.168.201.130@tcp:35:createmany.0 00000002:00100000:2.0:1788201388.785695:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff90235268b100 x1875064442724480/t17179869198(17179869198) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201388.785794:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201388.785826:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201388.785845:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201388.785860:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201388.785869:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201388.786128:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201388.786132:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201388.786141:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:2.0:1788201388.786147:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201388.786157:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201388.786174:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201388.786180:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:2.0:1788201388.786186:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201388.786200:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:2.0:1788201388.786205:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-1, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff902354a69800, open_flags 8042, mode 81a4 opened 0 00000080:00200000:2.0:1788201388.786213:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-1, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:2.0:1788201388.786232:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-1,[0x200000bd1:0x40c:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:2.0:1788201388.786299:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201388.786308:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201388.786321:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201388.786329:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234518e680 x1875064442724992/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201388.786338:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201388.786361:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518e680 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442724992:192.168.201.130@tcp:101:createmany.0 00000100:00100000:2.0:1788201388.786516:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3e40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201388.795312:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518e680 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442724992:192.168.201.130@tcp:101:createmany.0 00010000:00010000:0.0:1788201388.795340:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.795364:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.795373:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.795386:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.795403:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.795417:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.795433:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.795441:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201388.795453:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff90234518e680 x1875064442724992/t17179869200(17179869200) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201388.795467:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90234518e680 x1875064442724992/t17179869200(17179869200) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201388.795479:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201388.795491:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.795501:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1036 (0 0) 00000080:00200000:0.0:1788201388.795508:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201388.795521:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-1, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:0.0:1788201388.795551:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x40c:0x0](ffff9023449b5348) 00000080:00200000:0.0:1788201388.795675:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x40c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201388.795686:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x40c:0x0](ffff9023449b5348) 00000080:00010000:0.0:1788201388.795690:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x40c:0x0](ffff9023449b5348) for lock 0x4605c7c97e1c537a 00000080:00010000:0.0:1788201388.795700:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x40c:0x0](ffff9023449b5348) 00000080:00200008:0.0:1788201388.795713:0:17857:0:(file.c:1090:ll_file_open()) START file t-1:[0x200000bd1:0x40c:0x0](ffff9023449b5348), flags 100102 00000002:00100000:0.0:1788201388.795757:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90234518e680 x1875064442724992/t17179869200(17179869200) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201388.795797:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c537a from it ffff902342d497e0 00010000:00010000:0.0:1788201388.795802:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.795812:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201388.795833:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40c:0x0](ffff9023449b5348), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201388.795863:0:17857:0:(file.c:479:ll_file_release()) START file t-1:[0x200000bd1:0x40c:0x0](ffff9023449b5348), flags 100002 00000002:00100000:0.0:1788201388.795933:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff90234518e680 x1875064442724992/t17179869200(17179869200) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201388.795944:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201388.795950:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b0380 x1875064442725248/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201388.795970:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b0380 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442725248:192.168.201.130@tcp:35:createmany.0 00000100:00100000:0.0:1788201388.796027:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201388.804560:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b0380 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442725248:192.168.201.130@tcp:35:createmany.0 00000002:00100000:0.0:1788201388.804671:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff90234518e680 x1875064442724992/t17179869200(17179869200) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201388.804755:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201388.804773:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201388.804793:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201388.804815:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201388.804824:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201388.804831:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201388.804834:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201388.804842:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:0.0:1788201388.804846:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201388.804855:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201388.804870:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201388.805149:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:0.0:1788201388.805155:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201388.805178:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201388.805184:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-2, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff90235315c300, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201388.805192:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-2, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:0.0:1788201388.805210:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-2,[0x200000bd1:0x40d:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201388.805263:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.805271:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201388.805303:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201388.805312:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b0a80 x1875064442725504/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201388.805321:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201388.805343:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b0a80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442725504:192.168.201.130@tcp:101:createmany.0 00000100:00100000:0.0:1788201388.805501:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201388.811645:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b0a80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442725504:192.168.201.130@tcp:101:createmany.0 00010000:00010000:0.0:1788201388.811674:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.811697:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.811704:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.811714:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.811724:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.811738:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.811747:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.811755:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201388.811768:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023488b0a80 x1875064442725504/t17179869202(17179869202) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201388.811783:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b0a80 x1875064442725504/t17179869202(17179869202) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201388.811798:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201388.811811:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.811821:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1037 (0 0) 00000080:00200000:0.0:1788201388.811829:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201388.811844:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-2, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:1.0:1788201388.815850:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x40d:0x0](ffff9023449a6c08) 00000080:00200000:1.0:1788201388.816002:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x40d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201388.816016:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x40d:0x0](ffff9023449a6c08) 00000080:00010000:1.0:1788201388.816020:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x40d:0x0](ffff9023449a6c08) for lock 0x4605c7c97e1c5381 00000080:00010000:1.0:1788201388.816034:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x40d:0x0](ffff9023449a6c08) 00000080:00200008:1.0:1788201388.816051:0:17857:0:(file.c:1090:ll_file_open()) START file t-2:[0x200000bd1:0x40d:0x0](ffff9023449a6c08), flags 100102 00000002:00100000:1.0:1788201388.816083:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b0a80 x1875064442725504/t17179869202(17179869202) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201388.816130:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5381 from it ffff902353388a20 00010000:00010000:1.0:1788201388.816138:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.816160:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201388.816185:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40d:0x0](ffff9023449a6c08), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201388.816217:0:17857:0:(file.c:479:ll_file_release()) START file t-2:[0x200000bd1:0x40d:0x0](ffff9023449a6c08), flags 100002 00000002:00100000:1.0:1788201388.817425:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b0a80 x1875064442725504/t17179869202(17179869202) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201388.817441:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201388.817447:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458bbb80 x1875064442725760/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201388.817643:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458bbb80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442725760:192.168.201.130@tcp:35:createmany.0 00000100:00100000:1.0:1788201388.817826:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5ba80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201388.824388:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458bbb80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442725760:192.168.201.130@tcp:35:createmany.0 00000002:00100000:1.0:1788201388.824424:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b0a80 x1875064442725504/t17179869202(17179869202) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201388.824518:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201388.824537:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201388.824555:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201388.824581:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201388.824589:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201388.824597:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201388.824600:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201388.824608:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:1.0:1788201388.824612:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201388.824621:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201388.824636:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201388.824643:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:1.0:1788201388.824649:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201388.824661:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201388.824666:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-3, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff9023584e3c80, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201388.824674:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-3, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:1.0:1788201388.824692:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-3,[0x200000bd1:0x40e:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201388.824742:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.824750:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201388.824760:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201388.824767:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458b8a80 x1875064442726016/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201388.824775:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201388.824792:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b8a80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442726016:192.168.201.130@tcp:101:createmany.0 00000100:00100000:1.0:1788201388.824985:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5ba80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201388.834633:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b8a80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442726016:192.168.201.130@tcp:101:createmany.0 00010000:00010000:1.0:1788201388.834664:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.834689:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.834698:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.834710:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.834722:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.834736:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.834745:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.834753:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201388.834765:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023458b8a80 x1875064442726016/t17179869204(17179869204) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201388.834779:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458b8a80 x1875064442726016/t17179869204(17179869204) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201388.834793:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201388.834806:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.834817:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1038 (0 0) 00000080:00200000:1.0:1788201388.834824:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201388.834838:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-3, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:1.0:1788201388.834901:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x40e:0x0](ffff9023449a63c8) 00000080:00200000:1.0:1788201388.835019:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x40e:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201388.835030:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x40e:0x0](ffff9023449a63c8) 00000080:00010000:1.0:1788201388.835034:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x40e:0x0](ffff9023449a63c8) for lock 0x4605c7c97e1c5388 00000080:00010000:1.0:1788201388.835044:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x40e:0x0](ffff9023449a63c8) 00000080:00200008:1.0:1788201388.835057:0:17857:0:(file.c:1090:ll_file_open()) START file t-3:[0x200000bd1:0x40e:0x0](ffff9023449a63c8), flags 100102 00000002:00100000:1.0:1788201388.835076:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458b8a80 x1875064442726016/t17179869204(17179869204) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201388.835114:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5388 from it ffff9023450e7cc0 00010000:00010000:1.0:1788201388.835119:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.835129:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201388.835155:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40e:0x0](ffff9023449a63c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201388.835182:0:17857:0:(file.c:479:ll_file_release()) START file t-3:[0x200000bd1:0x40e:0x0](ffff9023449a63c8), flags 100002 00000002:00100000:1.0:1788201388.835218:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023458b8a80 x1875064442726016/t17179869204(17179869204) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201388.835229:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201388.835234:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458bb480 x1875064442726272/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201388.835255:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458bb480 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442726272:192.168.201.130@tcp:35:createmany.0 00000100:00100000:1.0:1788201388.838868:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201388.843769:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458bb480 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442726272:192.168.201.130@tcp:35:createmany.0 00000002:00100000:1.0:1788201388.843808:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023458b8a80 x1875064442726016/t17179869204(17179869204) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201388.844006:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201388.846669:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201388.846696:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201388.846713:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201388.846721:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201388.846730:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201388.846734:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201388.846742:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:0.0:1788201388.846747:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201388.846755:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788201388.846774:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201388.846784:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:0.0:1788201388.846792:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201388.846810:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201388.846816:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-4, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff9023584e3380, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201388.846824:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-4, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:0.0:1788201388.846846:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-4,[0x200000bd1:0x40f:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201388.849240:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.849258:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201388.849274:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201388.849296:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b2d80 x1875064442726528/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201388.849636:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201388.849662:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b2d80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442726528:192.168.201.130@tcp:101:createmany.0 00000100:00100000:0.0:1788201388.850130:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201388.865499:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b2d80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442726528:192.168.201.130@tcp:101:createmany.0 00010000:00010000:0.0:1788201388.865533:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.865560:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.865580:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.865591:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.865603:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.865618:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.865627:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.865636:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201388.865648:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023488b2d80 x1875064442726528/t17179869206(17179869206) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201388.865664:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b2d80 x1875064442726528/t17179869206(17179869206) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201388.865679:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201388.865694:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.865705:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1039 (0 0) 00000080:00200000:0.0:1788201388.865712:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201388.865726:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-4, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:0.0:1788201388.865756:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x40f:0x0](ffff9023449b0908) 00000080:00200000:0.0:1788201388.865922:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x40f:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201388.865933:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x40f:0x0](ffff9023449b0908) 00000080:00010000:0.0:1788201388.865937:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x40f:0x0](ffff9023449b0908) for lock 0x4605c7c97e1c538f 00000080:00010000:0.0:1788201388.865946:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x40f:0x0](ffff9023449b0908) 00000080:00200008:0.0:1788201388.865961:0:17857:0:(file.c:1090:ll_file_open()) START file t-4:[0x200000bd1:0x40f:0x0](ffff9023449b0908), flags 100102 00000002:00100000:0.0:1788201388.865981:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b2d80 x1875064442726528/t17179869206(17179869206) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201388.866023:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c538f from it ffff902353388000 00010000:00010000:0.0:1788201388.866028:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.866038:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201388.866066:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40f:0x0](ffff9023449b0908), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201388.866095:0:17857:0:(file.c:479:ll_file_release()) START file t-4:[0x200000bd1:0x40f:0x0](ffff9023449b0908), flags 100002 00000002:00100000:0.0:1788201388.866134:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b2d80 x1875064442726528/t17179869206(17179869206) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201388.866144:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201388.866150:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b1f80 x1875064442726784/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201388.866183:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1f80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442726784:192.168.201.130@tcp:35:createmany.0 00000100:00100000:0.0:1788201388.866271:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0e40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201388.870737:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1f80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442726784:192.168.201.130@tcp:35:createmany.0 00000002:00100000:0.0:1788201388.870773:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b2d80 x1875064442726528/t17179869206(17179869206) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201388.870858:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201388.870933:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201388.870953:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201388.870968:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788201388.870977:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201388.870983:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201388.870987:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201388.870995:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:0.0:1788201388.871000:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1788201388.871011:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:0.0:1788201388.871027:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201388.871033:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:0.0:1788201388.871039:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201388.871051:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201388.871056:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-5, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff90235315c600, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201388.871064:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-5, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:0.0:1788201388.871081:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-5,[0x200000bd1:0x410:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201388.871124:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.871133:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201388.871144:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201388.871151:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b1c00 x1875064442727040/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201388.871172:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201388.871192:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1c00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442727040:192.168.201.130@tcp:101:createmany.0 00000100:00100000:0.0:1788201388.871341:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0e40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201388.880619:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1c00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442727040:192.168.201.130@tcp:101:createmany.0 00010000:00010000:0.0:1788201388.880647:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.880670:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.880680:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.880692:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.880702:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.880716:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.880724:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.880732:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201388.880745:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023488b1c00 x1875064442727040/t17179869208(17179869208) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201388.880759:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b1c00 x1875064442727040/t17179869208(17179869208) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201388.880775:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201388.880788:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.880800:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1040 (0 0) 00000080:00200000:0.0:1788201388.880808:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201388.880826:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-5, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:0.0:1788201388.880857:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x410:0x0](ffff9023449b63c8) 00000080:00200000:2.0:1788201388.881336:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x410:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788201388.881355:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x410:0x0](ffff9023449b63c8) 00000080:00010000:2.0:1788201388.881359:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x410:0x0](ffff9023449b63c8) for lock 0x4605c7c97e1c5396 00000080:00010000:2.0:1788201388.881374:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x410:0x0](ffff9023449b63c8) 00000080:00200008:2.0:1788201388.881402:0:17857:0:(file.c:1090:ll_file_open()) START file t-5:[0x200000bd1:0x410:0x0](ffff9023449b63c8), flags 100102 00000002:00100000:2.0:1788201388.881426:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b1c00 x1875064442727040/t17179869208(17179869208) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788201388.881483:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5396 from it ffff902353388000 00010000:00010000:2.0:1788201388.881489:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201388.881500:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201388.881524:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x410:0x0](ffff9023449b63c8), inode mode 81a4 mask 40 00000080:00200008:2.0:1788201388.881563:0:17857:0:(file.c:479:ll_file_release()) START file t-5:[0x200000bd1:0x410:0x0](ffff9023449b63c8), flags 100002 00000002:00100000:2.0:1788201388.881614:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b1c00 x1875064442727040/t17179869208(17179869208) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201388.881626:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201388.881632:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234518dc00 x1875064442727296/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201388.881654:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518dc00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442727296:192.168.201.130@tcp:35:createmany.0 00000100:00100000:2.0:1788201388.881866:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe36c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201388.892730:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518dc00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442727296:192.168.201.130@tcp:35:createmany.0 00000002:00100000:3.0:1788201388.892758:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b1c00 x1875064442727040/t17179869208(17179869208) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201388.892833:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201388.892852:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201388.892867:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788201388.892908:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788201388.892914:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201388.892919:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201388.893085:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:3.0:1788201388.893092:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:3.0:1788201388.893095:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788201388.893103:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:3.0:1788201388.893123:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201388.893129:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:3.0:1788201388.893133:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:3.0:1788201388.893145:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:3.0:1788201388.893148:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-6, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff902343d63500, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788201388.893155:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-6, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:3.0:1788201388.893169:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-6,[0x200000bd1:0x411:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201388.893208:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.893213:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201388.893221:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201388.893226:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90235268a300 x1875064442727552/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201388.893233:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201388.893247:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442727552:192.168.201.130@tcp:101:createmany.0 00000100:00100000:3.0:1788201388.893391:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023584489c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201388.904975:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442727552:192.168.201.130@tcp:101:createmany.0 00010000:00010000:3.0:1788201388.905007:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.905032:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.905042:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.905053:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.905065:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.905084:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.905092:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.905100:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201388.905112:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff90235268a300 x1875064442727552/t17179869210(17179869210) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201388.905125:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90235268a300 x1875064442727552/t17179869210(17179869210) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201388.905138:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201388.905149:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.905166:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1041 (0 0) 00000080:00200000:3.0:1788201388.905172:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201388.905186:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-6, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:3.0:1788201388.905217:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x411:0x0](ffff90234497aa08) 00000080:00200000:3.0:1788201388.905352:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x411:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201388.905363:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x411:0x0](ffff90234497aa08) 00000080:00010000:3.0:1788201388.905366:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x411:0x0](ffff90234497aa08) for lock 0x4605c7c97e1c539d 00000080:00010000:3.0:1788201388.905375:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x411:0x0](ffff90234497aa08) 00000080:00200008:3.0:1788201388.905389:0:17857:0:(file.c:1090:ll_file_open()) START file t-6:[0x200000bd1:0x411:0x0](ffff90234497aa08), flags 100102 00000002:00100000:3.0:1788201388.905407:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90235268a300 x1875064442727552/t17179869210(17179869210) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201388.905446:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c539d from it ffff9023533eade0 00010000:00010000:3.0:1788201388.905451:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.905460:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201388.905481:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x411:0x0](ffff90234497aa08), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201388.905512:0:17857:0:(file.c:479:ll_file_release()) START file t-6:[0x200000bd1:0x411:0x0](ffff90234497aa08), flags 100002 00000002:00100000:3.0:1788201388.905554:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff90235268a300 x1875064442727552/t17179869210(17179869210) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201388.905565:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201388.905569:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352688000 x1875064442727808/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201388.905591:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442727808:192.168.201.130@tcp:35:createmany.0 00000100:00100000:3.0:1788201388.905675:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448240 going to sleep for 11 seconds 00000100:00100000:3.0:1788201388.911610:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442727808:192.168.201.130@tcp:35:createmany.0 00000002:00100000:3.0:1788201388.911646:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff90235268a300 x1875064442727552/t17179869210(17179869210) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201388.911727:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201388.911746:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201388.911765:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788201388.911779:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:3.0:1788201388.911787:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201388.911794:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201388.911798:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:3.0:1788201388.911807:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:3.0:1788201388.911811:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788201388.911820:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:3.0:1788201388.911837:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201388.911845:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:3.0:1788201388.911851:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:3.0:1788201388.911864:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:3.0:1788201388.911867:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-7, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff902343d62480, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788201388.911928:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-7, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:3.0:1788201388.911948:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-7,[0x200000bd1:0x412:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201388.912008:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.912016:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201388.912027:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201388.912033:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352689500 x1875064442728064/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201388.912041:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201388.912061:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442728064:192.168.201.130@tcp:101:createmany.0 00000100:00100000:3.0:1788201388.912188:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358448240 going to sleep for 11 seconds 00000100:00100000:3.0:1788201388.920805:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442728064:192.168.201.130@tcp:101:createmany.0 00010000:00010000:3.0:1788201388.920835:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.920858:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.920868:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.920913:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.920927:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.920944:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.920954:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201388.921679:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201388.921701:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff902352689500 x1875064442728064/t17179869212(17179869212) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201388.921715:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902352689500 x1875064442728064/t17179869212(17179869212) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201388.921728:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201388.927218:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201388.927263:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1042 (0 0) 00000080:00200000:2.0:1788201388.927273:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201388.927316:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-7, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:2.0:1788201388.927354:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x412:0x0](ffff9023449ac2c8) 00000080:00200000:2.0:1788201388.927568:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x412:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788201388.927578:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x412:0x0](ffff9023449ac2c8) 00000080:00010000:2.0:1788201388.927583:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x412:0x0](ffff9023449ac2c8) for lock 0x4605c7c97e1c53a4 00000080:00010000:2.0:1788201388.927593:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x412:0x0](ffff9023449ac2c8) 00000080:00200008:2.0:1788201388.927607:0:17857:0:(file.c:1090:ll_file_open()) START file t-7:[0x200000bd1:0x412:0x0](ffff9023449ac2c8), flags 100102 00000002:00100000:2.0:1788201388.927637:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902352689500 x1875064442728064/t17179869212(17179869212) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788201388.927679:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53a4 from it ffff9023533eade0 00010000:00010000:2.0:1788201388.927684:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201388.927695:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201388.927719:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x412:0x0](ffff9023449ac2c8), inode mode 81a4 mask 40 00000080:00200008:2.0:1788201388.927750:0:17857:0:(file.c:479:ll_file_release()) START file t-7:[0x200000bd1:0x412:0x0](ffff9023449ac2c8), flags 100002 00000002:00100000:2.0:1788201388.927799:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff902352689500 x1875064442728064/t17179869212(17179869212) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201388.927811:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201388.927816:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234518c700 x1875064442728320/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201388.927839:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518c700 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442728320:192.168.201.130@tcp:35:createmany.0 00000100:00100000:2.0:1788201388.930514:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201388.936461:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518c700 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442728320:192.168.201.130@tcp:35:createmany.0 00000002:00100000:0.0:1788201388.936501:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff902352689500 x1875064442728064/t17179869212(17179869212) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201388.936585:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201388.936603:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201388.936622:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:0.0:1788201388.936637:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:0.0:1788201388.936646:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201388.936652:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201388.936656:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201388.936663:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:0.0:1788201388.936668:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:0.0:1788201388.936678:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:0.0:1788201388.936693:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201388.936699:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:0.0:1788201388.936705:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201388.936717:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201388.936722:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-8, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff90235315d500, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201388.936729:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-8, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:0.0:1788201388.936747:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-8,[0x200000bd1:0x413:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201388.936791:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.936800:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201388.936812:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201388.936818:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b3800 x1875064442728576/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201388.936828:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201388.936847:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b3800 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442728576:192.168.201.130@tcp:101:createmany.0 00000100:00100000:0.0:1788201388.937057:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0e40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201388.949208:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b3800 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442728576:192.168.201.130@tcp:101:createmany.0 00010000:00010000:0.0:1788201388.949241:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.949263:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.949272:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.949293:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.949304:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.949316:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.949325:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.949333:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201388.949346:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023488b3800 x1875064442728576/t17179869214(17179869214) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201388.949359:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b3800 x1875064442728576/t17179869214(17179869214) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201388.949374:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201388.949386:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.949397:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1043 (0 0) 00000080:00200000:0.0:1788201388.949476:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201388.949491:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-8, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:0.0:1788201388.949523:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x413:0x0](ffff9023449b7448) 00000080:00200000:0.0:1788201388.949650:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x413:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201388.949661:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x413:0x0](ffff9023449b7448) 00000080:00010000:0.0:1788201388.949665:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x413:0x0](ffff9023449b7448) for lock 0x4605c7c97e1c53ab 00000080:00010000:0.0:1788201388.949676:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x413:0x0](ffff9023449b7448) 00000080:00200008:0.0:1788201388.949690:0:17857:0:(file.c:1090:ll_file_open()) START file t-8:[0x200000bd1:0x413:0x0](ffff9023449b7448), flags 100102 00000002:00100000:0.0:1788201388.949708:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b3800 x1875064442728576/t17179869214(17179869214) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201388.949748:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53ab from it ffff902353388540 00010000:00010000:0.0:1788201388.949754:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201388.949764:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201388.949788:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x413:0x0](ffff9023449b7448), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201388.949824:0:17857:0:(file.c:479:ll_file_release()) START file t-8:[0x200000bd1:0x413:0x0](ffff9023449b7448), flags 100002 00000002:00100000:0.0:1788201388.949949:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b3800 x1875064442728576/t17179869214(17179869214) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201388.949960:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201388.950030:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b0e00 x1875064442728832/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201388.950051:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b0e00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442728832:192.168.201.130@tcp:35:createmany.0 00000100:00100000:0.0:1788201388.950113:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0480 going to sleep for 11 seconds 00000100:00100000:0.0:1788201388.953714:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b0e00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442728832:192.168.201.130@tcp:35:createmany.0 00000002:00100000:0.0:1788201388.953749:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b3800 x1875064442728576/t17179869214(17179869214) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201388.953828:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201388.953847:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201388.954082:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788201388.954158:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788201388.954180:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201388.954189:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201388.954193:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201388.954202:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:1.0:1788201388.954206:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788201388.954216:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788201388.954234:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201388.954242:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:1.0:1788201388.954247:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201388.954260:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201388.954265:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-9, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff90235315cf00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201388.954273:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-9, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:1.0:1788201388.954302:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-9,[0x200000bd1:0x414:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201388.954359:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.954368:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201388.954389:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201388.954398:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458b8700 x1875064442729088/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201388.954408:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201388.954432:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b8700 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442729088:192.168.201.130@tcp:101:createmany.0 00000100:00100000:1.0:1788201388.954586:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201388.961445:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b8700 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442729088:192.168.201.130@tcp:101:createmany.0 00010000:00010000:1.0:1788201388.961475:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.961497:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.961506:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.961517:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.961527:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.961542:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.961550:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.961557:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201388.961569:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023458b8700 x1875064442729088/t17179869216(17179869216) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201388.961583:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458b8700 x1875064442729088/t17179869216(17179869216) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201388.961597:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201388.961610:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.961622:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1044 (0 0) 00000080:00200000:1.0:1788201388.961628:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201388.961642:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-9, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:1.0:1788201388.961669:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x414:0x0](ffff9023449a7448) 00000080:00200000:1.0:1788201388.961789:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x414:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201388.961800:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x414:0x0](ffff9023449a7448) 00000080:00010000:1.0:1788201388.961804:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x414:0x0](ffff9023449a7448) for lock 0x4605c7c97e1c53b2 00000080:00010000:1.0:1788201388.961814:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x414:0x0](ffff9023449a7448) 00000080:00200008:1.0:1788201388.961826:0:17857:0:(file.c:1090:ll_file_open()) START file t-9:[0x200000bd1:0x414:0x0](ffff9023449a7448), flags 100102 00000002:00100000:1.0:1788201388.961846:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458b8700 x1875064442729088/t17179869216(17179869216) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201388.961923:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53b2 from it ffff9023450e7cc0 00010000:00010000:1.0:1788201388.961928:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.961937:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201388.961960:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x414:0x0](ffff9023449a7448), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201388.961991:0:17857:0:(file.c:479:ll_file_release()) START file t-9:[0x200000bd1:0x414:0x0](ffff9023449a7448), flags 100002 00000002:00100000:1.0:1788201388.962034:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023458b8700 x1875064442729088/t17179869216(17179869216) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201388.962044:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201388.962048:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458b9500 x1875064442729344/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201388.962074:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9500 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442729344:192.168.201.130@tcp:35:createmany.0 00000100:00100000:1.0:1788201388.962303:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201388.965760:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9500 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442729344:192.168.201.130@tcp:35:createmany.0 00000002:00100000:1.0:1788201388.965792:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023458b8700 x1875064442729088/t17179869216(17179869216) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201388.965994:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201388.966016:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201388.966035:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788201388.966050:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788201388.966059:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201388.966065:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201388.966069:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201388.966076:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:1.0:1788201388.966080:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788201388.966089:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788201388.966105:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201388.966112:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:1.0:1788201388.966117:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201388.966130:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201388.966134:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-10, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff9023584e3c80, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201388.966142:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-10, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:1.0:1788201388.966162:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-10,[0x200000bd1:0x415:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201388.966215:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.966223:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201388.966235:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201388.966242:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458b9f80 x1875064442729600/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201388.966250:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201388.966270:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9f80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442729600:192.168.201.130@tcp:101:createmany.0 00000100:00100000:1.0:1788201388.966524:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201388.972134:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9f80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442729600:192.168.201.130@tcp:101:createmany.0 00010000:00010000:1.0:1788201388.972162:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.972184:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.972193:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.972205:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.972216:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.972230:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.972239:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.972246:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201388.972256:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023458b9f80 x1875064442729600/t17179869218(17179869218) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201388.972268:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458b9f80 x1875064442729600/t17179869218(17179869218) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201388.972295:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201388.972309:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.972320:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1045 (0 0) 00000080:00200000:1.0:1788201388.972328:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201388.972342:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-10, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:1.0:1788201388.972373:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x415:0x0](ffff9023449a4b08) 00000080:00200000:1.0:1788201388.972507:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x415:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201388.972518:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x415:0x0](ffff9023449a4b08) 00000080:00010000:1.0:1788201388.972521:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x415:0x0](ffff9023449a4b08) for lock 0x4605c7c97e1c53b9 00000080:00010000:1.0:1788201388.972532:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x415:0x0](ffff9023449a4b08) 00000080:00200008:1.0:1788201388.972545:0:17857:0:(file.c:1090:ll_file_open()) START file t-10:[0x200000bd1:0x415:0x0](ffff9023449a4b08), flags 100102 00000002:00100000:1.0:1788201388.972564:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458b9f80 x1875064442729600/t17179869218(17179869218) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201388.972608:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53b9 from it ffff9023450e7cc0 00010000:00010000:1.0:1788201388.972614:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.972624:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201388.972649:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x415:0x0](ffff9023449a4b08), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201388.972679:0:17857:0:(file.c:479:ll_file_release()) START file t-10:[0x200000bd1:0x415:0x0](ffff9023449a4b08), flags 100002 00000002:00100000:1.0:1788201388.972718:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023458b9f80 x1875064442729600/t17179869218(17179869218) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201388.972728:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201388.972734:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458b8380 x1875064442729856/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201388.972754:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b8380 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442729856:192.168.201.130@tcp:35:createmany.0 00000100:00100000:1.0:1788201388.972834:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201388.978960:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b8380 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442729856:192.168.201.130@tcp:35:createmany.0 00000002:00100000:1.0:1788201388.978998:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023458b9f80 x1875064442729600/t17179869218(17179869218) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201388.979079:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201388.979099:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201388.979118:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788201388.979133:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788201388.979178:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201388.979184:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201388.979188:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201388.979195:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:1.0:1788201388.979200:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788201388.979209:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788201388.979226:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201388.979233:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:1.0:1788201388.979238:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201388.979250:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201388.979254:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-11, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff9023584e3e00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201388.979263:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-11, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:1.0:1788201388.979291:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-11,[0x200000bd1:0x416:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201388.979344:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.979353:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201388.979366:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201388.979373:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458b9c00 x1875064442730112/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201388.979393:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201388.979415:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9c00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442730112:192.168.201.130@tcp:101:createmany.0 00000100:00100000:1.0:1788201388.979574:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201388.983954:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9c00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442730112:192.168.201.130@tcp:101:createmany.0 00010000:00010000:1.0:1788201388.983983:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.984007:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.984016:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.984028:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.984045:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.984059:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.984068:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.984075:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201388.984088:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023458b9c00 x1875064442730112/t17179869220(17179869220) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201388.984101:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458b9c00 x1875064442730112/t17179869220(17179869220) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201388.984115:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201388.984128:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.984138:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1046 (0 0) 00000080:00200000:1.0:1788201388.984145:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201388.984158:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-11, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:1.0:1788201388.984189:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x416:0x0](ffff9023449a21c8) 00000080:00200000:1.0:1788201388.984318:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x416:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201388.984329:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x416:0x0](ffff9023449a21c8) 00000080:00010000:1.0:1788201388.984333:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x416:0x0](ffff9023449a21c8) for lock 0x4605c7c97e1c53c0 00000080:00010000:1.0:1788201388.984343:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x416:0x0](ffff9023449a21c8) 00000080:00200008:1.0:1788201388.984356:0:17857:0:(file.c:1090:ll_file_open()) START file t-11:[0x200000bd1:0x416:0x0](ffff9023449a21c8), flags 100102 00000002:00100000:1.0:1788201388.984377:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458b9c00 x1875064442730112/t17179869220(17179869220) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201388.984419:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53c0 from it ffff9023450e7cc0 00010000:00010000:1.0:1788201388.984424:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.984435:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201388.984456:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x416:0x0](ffff9023449a21c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201388.984487:0:17857:0:(file.c:479:ll_file_release()) START file t-11:[0x200000bd1:0x416:0x0](ffff9023449a21c8), flags 100002 00000002:00100000:1.0:1788201388.984530:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023458b9c00 x1875064442730112/t17179869220(17179869220) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201388.984541:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201388.984546:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458b8000 x1875064442730368/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201388.984566:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b8000 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442730368:192.168.201.130@tcp:35:createmany.0 00000100:00100000:1.0:1788201388.984641:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b600 going to sleep for 11 seconds 00000100:00100000:1.0:1788201388.988133:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b8000 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442730368:192.168.201.130@tcp:35:createmany.0 00000002:00100000:1.0:1788201388.988168:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023458b9c00 x1875064442730112/t17179869220(17179869220) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201388.988246:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201388.988263:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201388.988291:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1788201388.988306:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788201388.988314:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201388.988320:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201388.988324:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201388.988333:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:1.0:1788201388.988337:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1788201388.988346:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:1.0:1788201388.988362:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201388.988369:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:1.0:1788201388.988375:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201388.988447:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201388.988453:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-12, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff9023584e3c80, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201388.988462:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-12, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:1.0:1788201388.988480:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-12,[0x200000bd1:0x417:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201388.988531:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.988540:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201388.988552:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201388.988560:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458ba300 x1875064442730624/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201388.988570:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201388.988587:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458ba300 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442730624:192.168.201.130@tcp:101:createmany.0 00000100:00100000:1.0:1788201388.988713:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b600 going to sleep for 11 seconds 00000100:00100000:1.0:1788201388.997992:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458ba300 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442730624:192.168.201.130@tcp:101:createmany.0 00010000:00010000:1.0:1788201388.998032:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.998060:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.998220:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.998234:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.998247:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.998263:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.998271:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.998287:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201388.998303:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023458ba300 x1875064442730624/t17179869222(17179869222) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201388.998316:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458ba300 x1875064442730624/t17179869222(17179869222) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201388.998332:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201388.998346:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.998357:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1047 (0 0) 00000080:00200000:1.0:1788201388.998364:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201388.998379:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-12, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:1.0:1788201388.998422:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x417:0x0](ffff9023449a3248) 00000080:00200000:1.0:1788201388.998550:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x417:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201388.998562:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x417:0x0](ffff9023449a3248) 00000080:00010000:1.0:1788201388.998566:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x417:0x0](ffff9023449a3248) for lock 0x4605c7c97e1c53c7 00000080:00010000:1.0:1788201388.998576:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x417:0x0](ffff9023449a3248) 00000080:00200008:1.0:1788201388.998590:0:17857:0:(file.c:1090:ll_file_open()) START file t-12:[0x200000bd1:0x417:0x0](ffff9023449a3248), flags 100102 00000002:00100000:1.0:1788201388.998610:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458ba300 x1875064442730624/t17179869222(17179869222) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201388.998651:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53c7 from it ffff9023450e7cc0 00010000:00010000:1.0:1788201388.998656:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201388.998666:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201388.998689:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x417:0x0](ffff9023449a3248), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201388.998732:0:17857:0:(file.c:479:ll_file_release()) START file t-12:[0x200000bd1:0x417:0x0](ffff9023449a3248), flags 100002 00000002:00100000:1.0:1788201388.998777:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023458ba300 x1875064442730624/t17179869222(17179869222) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201388.998788:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201388.998794:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458b8e00 x1875064442730880/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201388.998817:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b8e00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442730880:192.168.201.130@tcp:35:createmany.0 00000100:00100000:1.0:1788201388.998941:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bb40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.002082:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b8e00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442730880:192.168.201.130@tcp:35:createmany.0 00000002:00100000:1.0:1788201389.002118:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023458ba300 x1875064442730624/t17179869222(17179869222) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201404 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201389.002205:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.002222:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.002239:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1788201389.002255:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:1.0:1788201389.002264:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.002269:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.002272:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.002289:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:1.0:1788201389.002294:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1788201389.002303:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:1.0:1788201389.002319:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.002327:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:1.0:1788201389.002332:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.002347:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201389.002352:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-13, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff9023584e3200, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201389.002360:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-13, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:1.0:1788201389.002380:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-13,[0x200000bd1:0x418:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201389.002558:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.002566:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201389.002579:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201389.002586:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458baa00 x1875064442731136/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201389.002595:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.002615:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458baa00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442731136:192.168.201.130@tcp:101:createmany.0 00000100:00100000:1.0:1788201389.002740:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bb40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.007585:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458baa00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442731136:192.168.201.130@tcp:101:createmany.0 00010000:00010000:1.0:1788201389.007617:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.007642:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.007651:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.007661:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.007673:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.007689:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.007704:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.007725:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201389.007739:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023458baa00 x1875064442731136/t17179869224(17179869224) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201389.007754:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458baa00 x1875064442731136/t17179869224(17179869224) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201389.007768:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201389.007782:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.007793:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1048 (0 0) 00000080:00200000:1.0:1788201389.007800:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201389.007814:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-13, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:1.0:1788201389.007846:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x418:0x0](ffff9023449a42c8) 00000080:00200000:1.0:1788201389.007995:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x418:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201389.008006:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x418:0x0](ffff9023449a42c8) 00000080:00010000:1.0:1788201389.008010:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x418:0x0](ffff9023449a42c8) for lock 0x4605c7c97e1c53ce 00000080:00010000:1.0:1788201389.008020:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x418:0x0](ffff9023449a42c8) 00000080:00200008:1.0:1788201389.008033:0:17857:0:(file.c:1090:ll_file_open()) START file t-13:[0x200000bd1:0x418:0x0](ffff9023449a42c8), flags 100102 00000002:00100000:1.0:1788201389.008052:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458baa00 x1875064442731136/t17179869224(17179869224) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201389.008105:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53ce from it ffff9023450e7cc0 00010000:00010000:1.0:1788201389.008110:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.008120:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.008143:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x418:0x0](ffff9023449a42c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201389.008171:0:17857:0:(file.c:479:ll_file_release()) START file t-13:[0x200000bd1:0x418:0x0](ffff9023449a42c8), flags 100002 00000002:00100000:1.0:1788201389.008210:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023458baa00 x1875064442731136/t17179869224(17179869224) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201389.008221:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201389.008227:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458ba680 x1875064442731392/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201389.008247:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458ba680 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442731392:192.168.201.130@tcp:35:createmany.0 00000100:00100000:1.0:1788201389.008331:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.012103:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458ba680 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442731392:192.168.201.130@tcp:35:createmany.0 00000002:00100000:1.0:1788201389.012139:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023458baa00 x1875064442731136/t17179869224(17179869224) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201389.012216:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.012236:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.012254:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1788201389.012274:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:1.0:1788201389.012292:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.012298:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.012301:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.012309:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:1.0:1788201389.012313:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1788201389.012321:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:1.0:1788201389.012336:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.012344:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:1.0:1788201389.012351:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.012364:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201389.012368:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-14, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff9023584e3e00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201389.012376:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-14, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:1.0:1788201389.012567:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-14,[0x200000bd1:0x419:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201389.012616:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.012625:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201389.012636:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201389.012643:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458b9880 x1875064442731648/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201389.012652:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.012672:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9880 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442731648:192.168.201.130@tcp:101:createmany.0 00000100:00100000:1.0:1788201389.012803:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.017585:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9880 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442731648:192.168.201.130@tcp:101:createmany.0 00010000:00010000:1.0:1788201389.017614:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.017636:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.017644:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.017655:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.017665:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.017678:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.017686:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.017694:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201389.017706:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023458b9880 x1875064442731648/t17179869226(17179869226) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201389.017732:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458b9880 x1875064442731648/t17179869226(17179869226) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201389.017748:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201389.017761:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.017772:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1049 (0 0) 00000080:00200000:1.0:1788201389.017779:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201389.017792:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-14, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:1.0:1788201389.017825:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x419:0x0](ffff9023449a3a88) 00000080:00200000:1.0:1788201389.017973:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x419:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201389.017985:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x419:0x0](ffff9023449a3a88) 00000080:00010000:1.0:1788201389.017989:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x419:0x0](ffff9023449a3a88) for lock 0x4605c7c97e1c53d5 00000080:00010000:1.0:1788201389.017999:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x419:0x0](ffff9023449a3a88) 00000080:00200008:1.0:1788201389.018012:0:17857:0:(file.c:1090:ll_file_open()) START file t-14:[0x200000bd1:0x419:0x0](ffff9023449a3a88), flags 100102 00000002:00100000:1.0:1788201389.018030:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458b9880 x1875064442731648/t17179869226(17179869226) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201389.018273:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53d5 from it ffff9023450e7cc0 00010000:00010000:1.0:1788201389.018287:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.018298:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.018320:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x419:0x0](ffff9023449a3a88), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201389.018351:0:17857:0:(file.c:479:ll_file_release()) START file t-14:[0x200000bd1:0x419:0x0](ffff9023449a3a88), flags 100002 00000002:00100000:1.0:1788201389.018740:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023458b9880 x1875064442731648/t17179869226(17179869226) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201389.018753:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201389.018759:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458bb800 x1875064442731904/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201389.018782:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458bb800 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442731904:192.168.201.130@tcp:35:createmany.0 00000100:00100000:1.0:1788201389.019481:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b180 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.024198:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458bb800 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442731904:192.168.201.130@tcp:35:createmany.0 00000002:00100000:1.0:1788201389.024233:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023458b9880 x1875064442731648/t17179869226(17179869226) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201389.024328:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.024348:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.024366:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1788201389.024382:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:1.0:1788201389.024398:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.024404:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.024408:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.024416:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:1.0:1788201389.024420:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1788201389.024431:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:1.0:1788201389.024448:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.024455:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:1.0:1788201389.024460:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.024470:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201389.024475:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-15, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff9023584e3c80, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201389.024483:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-15, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:1.0:1788201389.024501:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-15,[0x200000bd1:0x41a:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201389.024550:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.024559:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201389.024582:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201389.024701:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458b9180 x1875064442732160/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201389.024789:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.024811:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458b9180 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442732160:192.168.201.130@tcp:101:createmany.0 00000100:00100000:0.0:1788201389.025330:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b180 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.031237:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458b9180 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442732160:192.168.201.130@tcp:101:createmany.0 00010000:00010000:0.0:1788201389.031268:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.031304:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.031313:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.031329:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.031340:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.031354:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.031368:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.031375:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201389.031386:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023458b9180 x1875064442732160/t17179869228(17179869228) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201389.031399:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458b9180 x1875064442732160/t17179869228(17179869228) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201389.031412:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201389.031425:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.031434:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1050 (0 0) 00000080:00200000:0.0:1788201389.031441:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201389.031455:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-15, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:0.0:1788201389.031485:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x41a:0x0](ffff9023449b1148) 00000080:00200000:0.0:1788201389.031677:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x41a:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201389.031689:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x41a:0x0](ffff9023449b1148) 00000080:00010000:0.0:1788201389.031693:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x41a:0x0](ffff9023449b1148) for lock 0x4605c7c97e1c53dc 00000080:00010000:0.0:1788201389.031703:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x41a:0x0](ffff9023449b1148) 00000080:00200008:0.0:1788201389.032045:0:17857:0:(file.c:1090:ll_file_open()) START file t-15:[0x200000bd1:0x41a:0x0](ffff9023449b1148), flags 100102 00000002:00100000:0.0:1788201389.032142:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458b9180 x1875064442732160/t17179869228(17179869228) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201389.032185:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53dc from it ffff9023450e7cc0 00010000:00010000:0.0:1788201389.032189:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.032198:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201389.032223:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x41a:0x0](ffff9023449b1148), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201389.032254:0:17857:0:(file.c:479:ll_file_release()) START file t-15:[0x200000bd1:0x41a:0x0](ffff9023449b1148), flags 100002 00000002:00100000:0.0:1788201389.032305:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023458b9180 x1875064442732160/t17179869228(17179869228) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201389.032316:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201389.032322:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b2300 x1875064442732416/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201389.032344:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b2300 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442732416:192.168.201.130@tcp:35:createmany.0 00000100:00100000:0.0:1788201389.032506:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.036480:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b2300 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442732416:192.168.201.130@tcp:35:createmany.0 00000002:00100000:0.0:1788201389.036516:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023458b9180 x1875064442732160/t17179869228(17179869228) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201389.036596:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.036614:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.036633:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:0.0:1788201389.036649:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:0.0:1788201389.036657:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.036662:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.036666:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.036673:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:0.0:1788201389.036677:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:0.0:1788201389.036685:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:0.0:1788201389.036700:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.036707:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:0.0:1788201389.036713:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.036734:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.036739:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-16, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff90235315d500, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201389.036748:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-16, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:0.0:1788201389.036767:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-16,[0x200000bd1:0x41b:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201389.036817:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.036825:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201389.036836:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201389.036842:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b0700 x1875064442732672/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201389.036851:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201389.036871:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b0700 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442732672:192.168.201.130@tcp:101:createmany.0 00000100:00100000:0.0:1788201389.040939:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.040988:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b0700 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442732672:192.168.201.130@tcp:101:createmany.0 00010000:00010000:0.0:1788201389.041011:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.041033:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.041041:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.041053:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.041080:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.041093:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.041102:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.041110:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201389.041120:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023488b0700 x1875064442732672/t17179869230(17179869230) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201389.041135:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b0700 x1875064442732672/t17179869230(17179869230) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201389.041148:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201389.041230:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.041244:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1051 (0 0) 00000080:00200000:0.0:1788201389.041251:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201389.041567:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-16, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:0.0:1788201389.041598:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x41b:0x0](ffff9023449b00c8) 00000080:00200000:0.0:1788201389.041724:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x41b:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201389.041734:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x41b:0x0](ffff9023449b00c8) 00000080:00010000:0.0:1788201389.041738:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x41b:0x0](ffff9023449b00c8) for lock 0x4605c7c97e1c53e3 00000080:00010000:0.0:1788201389.041748:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x41b:0x0](ffff9023449b00c8) 00000080:00200008:0.0:1788201389.041764:0:17857:0:(file.c:1090:ll_file_open()) START file t-16:[0x200000bd1:0x41b:0x0](ffff9023449b00c8), flags 100102 00000002:00100000:0.0:1788201389.041783:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b0700 x1875064442732672/t17179869230(17179869230) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201389.041823:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53e3 from it ffff902353388540 00010000:00010000:0.0:1788201389.041828:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.041838:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201389.041861:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x41b:0x0](ffff9023449b00c8), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201389.041921:0:17857:0:(file.c:479:ll_file_release()) START file t-16:[0x200000bd1:0x41b:0x0](ffff9023449b00c8), flags 100002 00000002:00100000:0.0:1788201389.041966:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b0700 x1875064442732672/t17179869230(17179869230) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201389.041977:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201389.041982:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b2a00 x1875064442732928/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201389.042002:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b2a00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442732928:192.168.201.130@tcp:35:createmany.0 00000100:00100000:0.0:1788201389.042574:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.047223:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b2a00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442732928:192.168.201.130@tcp:35:createmany.0 00000002:00100000:0.0:1788201389.047260:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b0700 x1875064442732672/t17179869230(17179869230) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201389.047352:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.047371:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.047400:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201389.047416:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:0.0:1788201389.047423:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.047429:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.047432:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.047441:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:0.0:1788201389.047445:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788201389.047453:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:0.0:1788201389.047469:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.047476:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:0.0:1788201389.047481:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.047494:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.047499:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-17, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff90235315de00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201389.047506:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-17, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:0.0:1788201389.047524:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-17,[0x200000bd1:0x41c:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201389.047574:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.047583:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201389.047594:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201389.047601:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b1500 x1875064442733184/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201389.047611:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201389.047631:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1500 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442733184:192.168.201.130@tcp:101:createmany.0 00000100:00100000:0.0:1788201389.047775:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.057215:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1500 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442733184:192.168.201.130@tcp:101:createmany.0 00010000:00010000:0.0:1788201389.057242:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.057265:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.057274:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.057296:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.057308:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.057323:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.057333:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.057341:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201389.057354:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023488b1500 x1875064442733184/t17179869232(17179869232) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201389.057367:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b1500 x1875064442733184/t17179869232(17179869232) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201389.057391:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201389.057405:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.057416:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1052 (0 0) 00000080:00200000:0.0:1788201389.057423:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201389.057436:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-17, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:0.0:1788201389.057467:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x41c:0x0](ffff9023449b3248) 00000080:00200000:0.0:1788201389.057585:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x41c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201389.057596:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x41c:0x0](ffff9023449b3248) 00000080:00010000:0.0:1788201389.057600:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x41c:0x0](ffff9023449b3248) for lock 0x4605c7c97e1c53ea 00000080:00010000:0.0:1788201389.057610:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x41c:0x0](ffff9023449b3248) 00000080:00200008:0.0:1788201389.057622:0:17857:0:(file.c:1090:ll_file_open()) START file t-17:[0x200000bd1:0x41c:0x0](ffff9023449b3248), flags 100102 00000002:00100000:0.0:1788201389.057641:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b1500 x1875064442733184/t17179869232(17179869232) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201389.057681:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53ea from it ffff902353388540 00010000:00010000:0.0:1788201389.057686:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.057696:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201389.057717:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x41c:0x0](ffff9023449b3248), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201389.057744:0:17857:0:(file.c:479:ll_file_release()) START file t-17:[0x200000bd1:0x41c:0x0](ffff9023449b3248), flags 100002 00000002:00100000:0.0:1788201389.057782:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b1500 x1875064442733184/t17179869232(17179869232) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201389.057792:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201389.057798:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b1880 x1875064442733440/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201389.057818:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442733440:192.168.201.130@tcp:35:createmany.0 00000100:00100000:0.0:1788201389.057873:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.062411:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1880 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442733440:192.168.201.130@tcp:35:createmany.0 00000002:00100000:0.0:1788201389.062450:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b1500 x1875064442733184/t17179869232(17179869232) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201389.062531:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.062549:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.062566:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:0.0:1788201389.062580:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:0.0:1788201389.062589:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.062595:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.062599:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.062606:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:0.0:1788201389.062611:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:0.0:1788201389.062620:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:0.0:1788201389.062636:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.062644:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:0.0:1788201389.062649:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.062664:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.062669:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-18, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff90235315d500, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201389.062677:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-18, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:0.0:1788201389.062694:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-18,[0x200000bd1:0x41d:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201389.062756:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.062765:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201389.062777:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201389.062784:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b1180 x1875064442733696/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201389.062797:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201389.062818:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b1180 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442733696:192.168.201.130@tcp:101:createmany.0 00000100:00100000:0.0:1788201389.063040:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.069857:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b1180 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442733696:192.168.201.130@tcp:101:createmany.0 00010000:00010000:0.0:1788201389.069912:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.070017:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.070028:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.070042:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.070055:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.070082:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.070092:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.070100:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201389.070113:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023488b1180 x1875064442733696/t17179869234(17179869234) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201389.070135:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b1180 x1875064442733696/t17179869234(17179869234) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201389.070149:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201389.070171:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.070183:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1053 (0 0) 00000080:00200000:1.0:1788201389.070191:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201389.070205:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-18, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:1.0:1788201389.070235:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x41d:0x0](ffff9023449a00c8) 00000080:00200000:1.0:1788201389.070378:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x41d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201389.070563:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x41d:0x0](ffff9023449a00c8) 00000080:00010000:1.0:1788201389.070567:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x41d:0x0](ffff9023449a00c8) for lock 0x4605c7c97e1c53f1 00000080:00010000:1.0:1788201389.070578:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x41d:0x0](ffff9023449a00c8) 00000080:00200008:1.0:1788201389.070593:0:17857:0:(file.c:1090:ll_file_open()) START file t-18:[0x200000bd1:0x41d:0x0](ffff9023449a00c8), flags 100102 00000002:00100000:1.0:1788201389.070613:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b1180 x1875064442733696/t17179869234(17179869234) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201389.070651:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53f1 from it ffff902353388540 00010000:00010000:1.0:1788201389.070656:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.070666:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.070698:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x41d:0x0](ffff9023449a00c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201389.070743:0:17857:0:(file.c:479:ll_file_release()) START file t-18:[0x200000bd1:0x41d:0x0](ffff9023449a00c8), flags 100002 00000002:00100000:1.0:1788201389.070794:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b1180 x1875064442733696/t17179869234(17179869234) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201389.070807:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201389.070813:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458bb100 x1875064442733952/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201389.070839:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458bb100 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442733952:192.168.201.130@tcp:35:createmany.0 00000100:00100000:1.0:1788201389.071005:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b780 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.078019:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458bb100 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442733952:192.168.201.130@tcp:35:createmany.0 00000002:00100000:1.0:1788201389.078075:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b1180 x1875064442733696/t17179869234(17179869234) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201389.078158:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.078302:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.078322:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1788201389.078337:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:1.0:1788201389.078346:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.078352:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.078356:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.078363:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:1.0:1788201389.078368:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1788201389.078387:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:1.0:1788201389.078405:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.078412:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:1.0:1788201389.078417:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.078431:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201389.078435:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-19, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff9023584e3e00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201389.078443:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-19, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:1.0:1788201389.078462:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-19,[0x200000bd1:0x41e:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201389.078510:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.078519:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201389.078531:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201389.078538:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023458bad80 x1875064442734208/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201389.078549:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.078568:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023458bad80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442734208:192.168.201.130@tcp:101:createmany.0 00000100:00100000:1.0:1788201389.078699:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b780 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.083244:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023458bad80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442734208:192.168.201.130@tcp:101:createmany.0 00010000:00010000:1.0:1788201389.083273:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.083305:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.083314:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.083325:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.083337:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.083353:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.083361:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.083369:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201389.083394:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023458bad80 x1875064442734208/t17179869236(17179869236) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201389.083409:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458bad80 x1875064442734208/t17179869236(17179869236) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201389.083425:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201389.083439:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.083450:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1054 (0 0) 00000080:00200000:1.0:1788201389.083458:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201389.083471:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-19, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:1.0:1788201389.083507:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x41e:0x0](ffff9023449a0908) 00000080:00200000:1.0:1788201389.083631:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x41e:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201389.083641:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x41e:0x0](ffff9023449a0908) 00000080:00010000:1.0:1788201389.083644:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x41e:0x0](ffff9023449a0908) for lock 0x4605c7c97e1c53f8 00000080:00010000:1.0:1788201389.083653:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x41e:0x0](ffff9023449a0908) 00000080:00200008:1.0:1788201389.083666:0:17857:0:(file.c:1090:ll_file_open()) START file t-19:[0x200000bd1:0x41e:0x0](ffff9023449a0908), flags 100102 00000002:00100000:1.0:1788201389.083684:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023458bad80 x1875064442734208/t17179869236(17179869236) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201389.083739:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53f8 from it ffff9023450e7060 00010000:00010000:1.0:1788201389.083745:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.083755:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.083777:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x41e:0x0](ffff9023449a0908), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201389.083805:0:17857:0:(file.c:479:ll_file_release()) START file t-19:[0x200000bd1:0x41e:0x0](ffff9023449a0908), flags 100002 00000002:00100000:1.0:1788201389.083844:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023458bad80 x1875064442734208/t17179869236(17179869236) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201389.083855:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201389.083861:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902345832680 x1875064442734464/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201389.083907:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345832680 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442734464:192.168.201.130@tcp:35:createmany.0 00000100:00100000:1.0:1788201389.083994:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bc00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.088662:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345832680 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442734464:192.168.201.130@tcp:35:createmany.0 00000002:00100000:1.0:1788201389.088724:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023458bad80 x1875064442734208/t17179869236(17179869236) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201389.088827:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.088847:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.088866:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1788201389.088917:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:1.0:1788201389.088927:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.088933:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.088937:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.088945:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:1.0:1788201389.088949:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1788201389.088958:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:1.0:1788201389.088975:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.088982:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:1.0:1788201389.088987:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.089000:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201389.089004:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-20, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff9023584e3200, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201389.089012:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-20, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:1.0:1788201389.089030:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-20,[0x200000bd1:0x41f:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201389.089092:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.089101:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201389.089112:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201389.089119:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902345833b80 x1875064442734720/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201389.089128:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.089147:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345833b80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442734720:192.168.201.130@tcp:101:createmany.0 00000100:00100000:1.0:1788201389.089289:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bc00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.106003:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345833b80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442734720:192.168.201.130@tcp:101:createmany.0 00010000:00010000:1.0:1788201389.106033:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.106057:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.106066:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.106077:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.106087:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.106101:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.106110:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.106117:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201389.106130:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff902345833b80 x1875064442734720/t17179869238(17179869238) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201389.106143:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902345833b80 x1875064442734720/t17179869238(17179869238) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201389.106157:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201389.106169:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.106179:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1055 (0 0) 00000080:00200000:1.0:1788201389.106185:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201389.106199:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-20, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:1.0:1788201389.106231:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x41f:0x0](ffff9023449a5348) 00000080:00200000:1.0:1788201389.106381:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x41f:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201389.106393:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x41f:0x0](ffff9023449a5348) 00000080:00010000:1.0:1788201389.106396:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x41f:0x0](ffff9023449a5348) for lock 0x4605c7c97e1c53ff 00000080:00010000:1.0:1788201389.106406:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x41f:0x0](ffff9023449a5348) 00000080:00200008:1.0:1788201389.106420:0:17857:0:(file.c:1090:ll_file_open()) START file t-20:[0x200000bd1:0x41f:0x0](ffff9023449a5348), flags 100102 00000002:00100000:1.0:1788201389.106439:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902345833b80 x1875064442734720/t17179869238(17179869238) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201389.106481:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53ff from it ffff9023450e7060 00010000:00010000:1.0:1788201389.106486:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.106497:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.106521:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x41f:0x0](ffff9023449a5348), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201389.106553:0:17857:0:(file.c:479:ll_file_release()) START file t-20:[0x200000bd1:0x41f:0x0](ffff9023449a5348), flags 100002 00000002:00100000:1.0:1788201389.106596:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff902345833b80 x1875064442734720/t17179869238(17179869238) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201389.106607:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201389.106612:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902345833100 x1875064442734976/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201389.106639:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345833100 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442734976:192.168.201.130@tcp:35:createmany.0 00000100:00100000:1.0:1788201389.107072:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5be40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.115211:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345833100 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442734976:192.168.201.130@tcp:35:createmany.0 00000002:00100000:1.0:1788201389.115253:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff902345833b80 x1875064442734720/t17179869238(17179869238) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201389.115349:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.115371:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.115441:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:1.0:1788201389.115458:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:1.0:1788201389.115467:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.115474:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.115478:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.115485:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:1.0:1788201389.115490:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:1.0:1788201389.115499:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:1.0:1788201389.115516:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.115523:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:1.0:1788201389.115529:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.115543:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201389.115548:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-21, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff9023584e3e00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201389.115558:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-21, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:1.0:1788201389.115579:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-21,[0x200000bd1:0x420:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201389.115630:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.115639:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201389.115649:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201389.115656:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902345831500 x1875064442735232/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201389.115666:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.115685:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345831500 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442735232:192.168.201.130@tcp:101:createmany.0 00000100:00100000:1.0:1788201389.116668:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5be40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.122264:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345831500 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442735232:192.168.201.130@tcp:101:createmany.0 00010000:00010000:1.0:1788201389.122303:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.122327:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.122336:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.122348:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.122360:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.122376:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.122384:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.122393:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201389.122406:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff902345831500 x1875064442735232/t17179869240(17179869240) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201389.122420:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902345831500 x1875064442735232/t17179869240(17179869240) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201389.122434:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201389.122446:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.122458:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1056 (0 0) 00000080:00200000:1.0:1788201389.122465:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201389.122479:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-21, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:1.0:1788201389.122510:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x420:0x0](ffff9023449a1148) 00000080:00200000:1.0:1788201389.122634:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x420:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201389.122645:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x420:0x0](ffff9023449a1148) 00000080:00010000:1.0:1788201389.122649:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x420:0x0](ffff9023449a1148) for lock 0x4605c7c97e1c5406 00000080:00010000:1.0:1788201389.122659:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x420:0x0](ffff9023449a1148) 00000080:00200008:1.0:1788201389.122673:0:17857:0:(file.c:1090:ll_file_open()) START file t-21:[0x200000bd1:0x420:0x0](ffff9023449a1148), flags 100102 00000002:00100000:1.0:1788201389.122691:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902345831500 x1875064442735232/t17179869240(17179869240) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201389.122732:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5406 from it ffff9023450e7060 00010000:00010000:1.0:1788201389.122737:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.122746:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.122768:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x420:0x0](ffff9023449a1148), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201389.122799:0:17857:0:(file.c:479:ll_file_release()) START file t-21:[0x200000bd1:0x420:0x0](ffff9023449a1148), flags 100002 00000002:00100000:1.0:1788201389.122838:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff902345831500 x1875064442735232/t17179869240(17179869240) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201389.122849:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201389.122855:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902345830700 x1875064442735488/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201389.122910:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345830700 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442735488:192.168.201.130@tcp:35:createmany.0 00000100:00100000:1.0:1788201389.122980:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.126340:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345830700 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442735488:192.168.201.130@tcp:35:createmany.0 00000002:00100000:1.0:1788201389.126372:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff902345831500 x1875064442735232/t17179869240(17179869240) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201389.126457:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.126474:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.126490:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:1.0:1788201389.126504:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:1.0:1788201389.126511:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.126516:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.126520:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.126526:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:1.0:1788201389.126530:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:1.0:1788201389.126539:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:1.0:1788201389.126553:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.126558:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:1.0:1788201389.126563:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.126586:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201389.126591:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-22, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff9023584e2d80, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201389.126597:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-22, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:1.0:1788201389.126615:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-22,[0x200000bd1:0x421:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201389.126661:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.126668:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201389.126679:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201389.126686:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902345833800 x1875064442735744/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201389.126695:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.126713:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345833800 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442735744:192.168.201.130@tcp:101:createmany.0 00000100:00100000:1.0:1788201389.126831:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.135981:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345833800 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442735744:192.168.201.130@tcp:101:createmany.0 00010000:00010000:1.0:1788201389.136011:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.136032:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.136039:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.136051:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.136069:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.136084:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.136093:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.136101:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201389.136112:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff902345833800 x1875064442735744/t17179869242(17179869242) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201389.136123:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902345833800 x1875064442735744/t17179869242(17179869242) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201389.136136:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201389.136148:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.136158:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1057 (0 0) 00000080:00200000:1.0:1788201389.136165:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201389.136179:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-22, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:1.0:1788201389.136210:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x421:0x0](ffff9023449a5b88) 00000080:00200000:1.0:1788201389.136347:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x421:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201389.136359:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x421:0x0](ffff9023449a5b88) 00000080:00010000:1.0:1788201389.136362:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x421:0x0](ffff9023449a5b88) for lock 0x4605c7c97e1c540d 00000080:00010000:1.0:1788201389.136372:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x421:0x0](ffff9023449a5b88) 00000080:00200008:1.0:1788201389.136385:0:17857:0:(file.c:1090:ll_file_open()) START file t-22:[0x200000bd1:0x421:0x0](ffff9023449a5b88), flags 100102 00000002:00100000:1.0:1788201389.136404:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902345833800 x1875064442735744/t17179869242(17179869242) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201389.136444:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c540d from it ffff9023450e7060 00010000:00010000:1.0:1788201389.136449:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.136458:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.136480:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x421:0x0](ffff9023449a5b88), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201389.136506:0:17857:0:(file.c:479:ll_file_release()) START file t-22:[0x200000bd1:0x421:0x0](ffff9023449a5b88), flags 100002 00000002:00100000:1.0:1788201389.136545:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff902345833800 x1875064442735744/t17179869242(17179869242) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201389.136556:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201389.136561:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902345831f80 x1875064442736000/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201389.136582:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345831f80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442736000:192.168.201.130@tcp:35:createmany.0 00000100:00100000:1.0:1788201389.136656:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bd80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.140018:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345831f80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442736000:192.168.201.130@tcp:35:createmany.0 00000002:00100000:0.0:1788201389.141670:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff902345833800 x1875064442735744/t17179869242(17179869242) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201389.143421:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.143449:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.143468:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:0.0:1788201389.143486:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:0.0:1788201389.143495:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.143503:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.143507:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.143515:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:0.0:1788201389.143520:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:0.0:1788201389.143531:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:0.0:1788201389.143549:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.143557:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:0.0:1788201389.143563:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.143587:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.143592:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-23, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff90235315de00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201389.143600:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-23, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:0.0:1788201389.143619:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-23,[0x200000bd1:0x422:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201389.143668:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.143677:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201389.143691:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201389.143702:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b3100 x1875064442736256/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201389.143712:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201389.143734:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b3100 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442736256:192.168.201.130@tcp:101:createmany.0 00000100:00100000:0.0:1788201389.143909:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.151196:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b3100 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442736256:192.168.201.130@tcp:101:createmany.0 00010000:00010000:0.0:1788201389.151224:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.151247:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.151256:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.151267:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.151286:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.151301:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.151309:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.151315:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201389.151328:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023488b3100 x1875064442736256/t17179869244(17179869244) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201389.151350:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b3100 x1875064442736256/t17179869244(17179869244) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201389.151364:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201389.151387:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.151400:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1058 (0 0) 00000080:00200000:0.0:1788201389.151406:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201389.151419:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-23, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:0.0:1788201389.151455:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x422:0x0](ffff9023449b42c8) 00000080:00200000:0.0:1788201389.151770:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x422:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201389.151782:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x422:0x0](ffff9023449b42c8) 00000080:00010000:0.0:1788201389.151786:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x422:0x0](ffff9023449b42c8) for lock 0x4605c7c97e1c5414 00000080:00010000:0.0:1788201389.151796:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x422:0x0](ffff9023449b42c8) 00000080:00200008:0.0:1788201389.151810:0:17857:0:(file.c:1090:ll_file_open()) START file t-23:[0x200000bd1:0x422:0x0](ffff9023449b42c8), flags 100102 00000002:00100000:0.0:1788201389.151834:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b3100 x1875064442736256/t17179869244(17179869244) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201389.151925:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5414 from it ffff9023533886c0 00010000:00010000:0.0:1788201389.151931:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.151943:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201389.151974:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x422:0x0](ffff9023449b42c8), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201389.152005:0:17857:0:(file.c:479:ll_file_release()) START file t-23:[0x200000bd1:0x422:0x0](ffff9023449b42c8), flags 100002 00000002:00100000:0.0:1788201389.152044:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b3100 x1875064442736256/t17179869244(17179869244) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201389.152054:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201389.152058:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b3480 x1875064442736512/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201389.152077:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b3480 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442736512:192.168.201.130@tcp:35:createmany.0 00000100:00100000:0.0:1788201389.152162:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.162017:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b3480 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442736512:192.168.201.130@tcp:35:createmany.0 00000002:00100000:0.0:1788201389.162054:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b3100 x1875064442736256/t17179869244(17179869244) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201389.162139:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.162160:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.162180:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:0.0:1788201389.162195:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:0.0:1788201389.162203:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.162209:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.162213:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.162221:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:0.0:1788201389.162226:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:0.0:1788201389.162235:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:0.0:1788201389.162251:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.162258:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:0.0:1788201389.162264:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.162292:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.162297:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-24, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff90235315d980, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201389.162306:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-24, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:0.0:1788201389.162326:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-24,[0x200000bd1:0x423:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201389.162453:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.162463:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201389.162476:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201389.162483:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b3b80 x1875064442736768/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201389.162495:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201389.162516:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b3b80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442736768:192.168.201.130@tcp:101:createmany.0 00000100:00100000:0.0:1788201389.162723:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.180335:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b3b80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442736768:192.168.201.130@tcp:101:createmany.0 00010000:00010000:0.0:1788201389.180367:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.180648:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.180658:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.180672:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.180684:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.180699:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.180708:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.180771:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201389.180785:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023488b3b80 x1875064442736768/t17179869246(17179869246) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201389.180800:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b3b80 x1875064442736768/t17179869246(17179869246) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201389.180818:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201389.180833:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.180846:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1059 (0 0) 00000080:00200000:0.0:1788201389.180855:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201389.181232:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-24, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:0.0:1788201389.181286:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x423:0x0](ffff9023449b2a08) 00000080:00200000:0.0:1788201389.181775:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x423:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201389.181789:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x423:0x0](ffff9023449b2a08) 00000080:00010000:0.0:1788201389.181793:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x423:0x0](ffff9023449b2a08) for lock 0x4605c7c97e1c541b 00000080:00010000:0.0:1788201389.181806:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x423:0x0](ffff9023449b2a08) 00000080:00200008:0.0:1788201389.181822:0:17857:0:(file.c:1090:ll_file_open()) START file t-24:[0x200000bd1:0x423:0x0](ffff9023449b2a08), flags 100102 00000002:00100000:0.0:1788201389.181846:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b3b80 x1875064442736768/t17179869246(17179869246) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201389.181914:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c541b from it ffff9023533886c0 00010000:00010000:0.0:1788201389.181919:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.181930:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201389.181957:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x423:0x0](ffff9023449b2a08), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201389.181997:0:17857:0:(file.c:479:ll_file_release()) START file t-24:[0x200000bd1:0x423:0x0](ffff9023449b2a08), flags 100002 00000002:00100000:0.0:1788201389.182046:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b3b80 x1875064442736768/t17179869246(17179869246) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201389.182057:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201389.182063:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b2680 x1875064442737024/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201389.182087:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b2680 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442737024:192.168.201.130@tcp:35:createmany.0 00000100:00100000:0.0:1788201389.182311:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe09c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.191795:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b2680 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442737024:192.168.201.130@tcp:35:createmany.0 00000002:00100000:0.0:1788201389.191832:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b3b80 x1875064442736768/t17179869246(17179869246) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201389.191979:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.192001:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.192024:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:0.0:1788201389.192039:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:0.0:1788201389.192047:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.192052:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.192056:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.192064:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:0.0:1788201389.192069:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:0.0:1788201389.192078:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:0.0:1788201389.192097:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.192104:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:0.0:1788201389.192110:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.192122:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.192126:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-25, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff90235315de00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201389.192135:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-25, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:0.0:1788201389.192152:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-25,[0x200000bd1:0x424:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201389.192214:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.192222:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201389.192232:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201389.192239:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023488b0000 x1875064442737280/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201389.192248:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201389.192267:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023488b0000 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442737280:192.168.201.130@tcp:101:createmany.0 00000100:00100000:0.0:1788201389.192424:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe09c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.200968:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023488b0000 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442737280:192.168.201.130@tcp:101:createmany.0 00010000:00010000:0.0:1788201389.200998:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.201021:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.201030:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.201040:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.201051:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.201067:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.201075:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.201083:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201389.201095:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023488b0000 x1875064442737280/t17179869248(17179869248) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201389.201107:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b0000 x1875064442737280/t17179869248(17179869248) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201389.201122:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201389.201135:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.201148:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1060 (0 0) 00000080:00200000:0.0:1788201389.201155:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201389.201168:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-25, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:0.0:1788201389.201199:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x424:0x0](ffff9023449b6c08) 00000080:00200000:0.0:1788201389.201323:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x424:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201389.201334:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x424:0x0](ffff9023449b6c08) 00000080:00010000:0.0:1788201389.201338:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x424:0x0](ffff9023449b6c08) for lock 0x4605c7c97e1c5422 00000080:00010000:0.0:1788201389.201348:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x424:0x0](ffff9023449b6c08) 00000080:00200008:0.0:1788201389.201360:0:17857:0:(file.c:1090:ll_file_open()) START file t-25:[0x200000bd1:0x424:0x0](ffff9023449b6c08), flags 100102 00000002:00100000:0.0:1788201389.201381:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023488b0000 x1875064442737280/t17179869248(17179869248) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201389.201421:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5422 from it ffff9023533886c0 00010000:00010000:0.0:1788201389.201426:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.201435:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201389.201457:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x424:0x0](ffff9023449b6c08), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201389.201486:0:17857:0:(file.c:479:ll_file_release()) START file t-25:[0x200000bd1:0x424:0x0](ffff9023449b6c08), flags 100002 00000002:00100000:0.0:1788201389.201530:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023488b0000 x1875064442737280/t17179869248(17179869248) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201389.201542:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201389.201547:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450bad80 x1875064442737536/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201389.201586:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442737536:192.168.201.130@tcp:35:createmany.0 00000100:00100000:0.0:1788201389.201658:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.210225:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442737536:192.168.201.130@tcp:35:createmany.0 00000002:00100000:0.0:1788201389.210260:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023488b0000 x1875064442737280/t17179869248(17179869248) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201389.210353:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.210375:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.210408:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:0.0:1788201389.210424:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:0.0:1788201389.210432:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.210439:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.210442:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.210450:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:0.0:1788201389.210454:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:0.0:1788201389.210462:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:0.0:1788201389.210480:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.210487:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:0.0:1788201389.210493:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.210508:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.210513:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-26, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff90235315d980, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201389.210521:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-26, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:0.0:1788201389.210541:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-26,[0x200000bd1:0x425:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201389.210862:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.211065:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201389.211078:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201389.211087:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450b9180 x1875064442737792/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201389.211097:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.212345:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442737792:192.168.201.130@tcp:101:createmany.0 00000100:00100000:1.0:1788201389.212681:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bd80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.222856:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442737792:192.168.201.130@tcp:101:createmany.0 00010000:00010000:1.0:1788201389.222974:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.223008:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.223018:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.223030:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.223043:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.223059:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.223070:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.223079:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201389.223098:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9023450b9180 x1875064442737792/t17179869250(17179869250) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201389.223115:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023450b9180 x1875064442737792/t17179869250(17179869250) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201389.223132:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201389.223149:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.223161:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1061 (0 0) 00000080:00200000:1.0:1788201389.223169:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201389.223190:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-26, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:1.0:1788201389.223233:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x425:0x0](ffff9023449a2a08) 00000080:00200000:1.0:1788201389.223851:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x425:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201389.223995:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x425:0x0](ffff9023449a2a08) 00000080:00010000:1.0:1788201389.224000:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x425:0x0](ffff9023449a2a08) for lock 0x4605c7c97e1c5429 00000080:00010000:1.0:1788201389.224011:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x425:0x0](ffff9023449a2a08) 00000080:00200008:1.0:1788201389.224025:0:17857:0:(file.c:1090:ll_file_open()) START file t-26:[0x200000bd1:0x425:0x0](ffff9023449a2a08), flags 100102 00000002:00100000:1.0:1788201389.224050:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023450b9180 x1875064442737792/t17179869250(17179869250) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201389.224089:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5429 from it ffff9023533886c0 00010000:00010000:1.0:1788201389.224095:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.224112:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.224137:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x425:0x0](ffff9023449a2a08), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201389.224168:0:17857:0:(file.c:479:ll_file_release()) START file t-26:[0x200000bd1:0x425:0x0](ffff9023449a2a08), flags 100002 00000002:00100000:1.0:1788201389.224214:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023450b9180 x1875064442737792/t17179869250(17179869250) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201389.224225:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201389.224230:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902345832d80 x1875064442738048/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201389.224251:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345832d80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442738048:192.168.201.130@tcp:35:createmany.0 00000100:00100000:1.0:1788201389.224425:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bf00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.231019:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345832d80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442738048:192.168.201.130@tcp:35:createmany.0 00000002:00100000:1.0:1788201389.231061:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023450b9180 x1875064442737792/t17179869250(17179869250) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201389.231151:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.231187:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.231208:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:1.0:1788201389.231224:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:1.0:1788201389.231233:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.231239:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.231244:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.231252:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:1.0:1788201389.231257:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:1.0:1788201389.231266:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:1.0:1788201389.231290:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.231298:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:1.0:1788201389.231303:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.231324:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201389.231331:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-27, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff9023584e3e00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201389.231339:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-27, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:1.0:1788201389.231361:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-27,[0x200000bd1:0x426:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201389.232375:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.232431:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201389.232442:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201389.232450:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902345831c00 x1875064442738304/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201389.232456:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.232475:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345831c00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442738304:192.168.201.130@tcp:101:createmany.0 00000100:00100000:1.0:1788201389.232716:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5bf00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.240728:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345831c00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442738304:192.168.201.130@tcp:101:createmany.0 00010000:00010000:1.0:1788201389.240756:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.240781:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.240790:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.240802:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.240813:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.240827:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.240836:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.240844:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201389.240855:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff902345831c00 x1875064442738304/t17179869252(17179869252) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201389.240914:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902345831c00 x1875064442738304/t17179869252(17179869252) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201389.240929:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201389.240941:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.240951:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1062 (0 0) 00000080:00200000:1.0:1788201389.240958:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201389.240971:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-27, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:1.0:1788201389.241000:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x426:0x0](ffff9023448e7448) 00000080:00200000:1.0:1788201389.241128:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x426:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201389.241141:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x426:0x0](ffff9023448e7448) 00000080:00010000:1.0:1788201389.241145:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x426:0x0](ffff9023448e7448) for lock 0x4605c7c97e1c5430 00000080:00010000:1.0:1788201389.241155:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x426:0x0](ffff9023448e7448) 00000080:00200008:1.0:1788201389.241172:0:17857:0:(file.c:1090:ll_file_open()) START file t-27:[0x200000bd1:0x426:0x0](ffff9023448e7448), flags 100102 00000002:00100000:1.0:1788201389.241194:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902345831c00 x1875064442738304/t17179869252(17179869252) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201389.241238:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5430 from it ffff9023450e7060 00010000:00010000:1.0:1788201389.241243:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.241252:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.241291:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x426:0x0](ffff9023448e7448), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201389.241322:0:17857:0:(file.c:479:ll_file_release()) START file t-27:[0x200000bd1:0x426:0x0](ffff9023448e7448), flags 100002 00000002:00100000:1.0:1788201389.241364:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff902345831c00 x1875064442738304/t17179869252(17179869252) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201389.241375:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201389.241380:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902345830380 x1875064442738560/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201389.241400:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345830380 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442738560:192.168.201.130@tcp:35:createmany.0 00000100:00100000:1.0:1788201389.241470:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b180 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.245361:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345830380 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442738560:192.168.201.130@tcp:35:createmany.0 00000002:00100000:1.0:1788201389.245464:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff902345831c00 x1875064442738304/t17179869252(17179869252) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201389.245551:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.245571:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.245589:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:1.0:1788201389.245603:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:1.0:1788201389.245611:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.245617:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.245620:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.245627:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:1.0:1788201389.245632:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:1.0:1788201389.245641:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:1.0:1788201389.245656:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.245663:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:1.0:1788201389.245669:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.245682:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201389.245687:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-28, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff9023584e3200, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201389.245695:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-28, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:1.0:1788201389.245789:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-28,[0x200000bd1:0x427:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201389.245841:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.245851:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201389.246133:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201389.246142:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902345831880 x1875064442738816/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201389.246151:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.246309:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345831880 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442738816:192.168.201.130@tcp:101:createmany.0 00000100:00100000:1.0:1788201389.246574:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b180 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.252379:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345831880 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442738816:192.168.201.130@tcp:101:createmany.0 00010000:00010000:1.0:1788201389.252415:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.252439:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.252449:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.252461:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.252473:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.252487:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.252496:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.252504:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201389.252516:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff902345831880 x1875064442738816/t17179869254(17179869254) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201389.252530:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902345831880 x1875064442738816/t17179869254(17179869254) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201389.252546:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201389.252559:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.252570:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1063 (0 0) 00000080:00200000:1.0:1788201389.252576:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201389.252588:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-28, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:1.0:1788201389.252618:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x427:0x0](ffff9023448e1988) 00000080:00200000:1.0:1788201389.252752:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x427:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201389.252763:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x427:0x0](ffff9023448e1988) 00000080:00010000:1.0:1788201389.252766:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x427:0x0](ffff9023448e1988) for lock 0x4605c7c97e1c5437 00000080:00010000:1.0:1788201389.252777:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x427:0x0](ffff9023448e1988) 00000080:00200008:1.0:1788201389.252791:0:17857:0:(file.c:1090:ll_file_open()) START file t-28:[0x200000bd1:0x427:0x0](ffff9023448e1988), flags 100102 00000002:00100000:1.0:1788201389.252808:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902345831880 x1875064442738816/t17179869254(17179869254) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201389.252846:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5437 from it ffff9023450e7060 00010000:00010000:1.0:1788201389.252850:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.252858:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.252931:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x427:0x0](ffff9023448e1988), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201389.252963:0:17857:0:(file.c:479:ll_file_release()) START file t-28:[0x200000bd1:0x427:0x0](ffff9023448e1988), flags 100002 00000002:00100000:1.0:1788201389.253005:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff902345831880 x1875064442738816/t17179869254(17179869254) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201389.253016:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201389.253022:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902345833480 x1875064442739072/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201389.253043:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345833480 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442739072:192.168.201.130@tcp:35:createmany.0 00000100:00100000:1.0:1788201389.253112:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358723d80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.257205:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345833480 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442739072:192.168.201.130@tcp:35:createmany.0 00000002:00100000:1.0:1788201389.257245:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff902345831880 x1875064442738816/t17179869254(17179869254) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201389.257344:0:17857:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.257366:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.257397:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1788201389.257413:0:17857:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:1.0:1788201389.257421:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.257427:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.257431:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.257438:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c53bf8 00010000:00010000:1.0:1788201389.257442:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1788201389.257452:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:1.0:1788201389.257469:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.257476:0:17857:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=25 00000080:00200000:1.0:1788201389.257482:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.257496:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201389.257500:0:17857:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=t-29, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), file ffff9023584e3e00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201389.257508:0:17857:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=t-29, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000002:00010000:1.0:1788201389.257527:0:17857:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: t-29,[0x200000bd1:0x428:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201389.257583:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.257590:0:17857:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201389.257601:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201389.257608:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902345830a80 x1875064442739328/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201389.257618:0:17857:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17857 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.257637:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345830a80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442739328:192.168.201.130@tcp:101:createmany.0 00000100:00100000:1.0:1788201389.257774:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358723d80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.264967:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345830a80 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442739328:192.168.201.130@tcp:101:createmany.0 00010000:00010000:1.0:1788201389.265000:0:17857:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.265022:0:17857:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.265031:0:17857:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.265043:0:17857:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.265054:0:17857:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.265067:0:17857:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.265075:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.265083:0:17857:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201389.265096:0:17857:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff902345830a80 x1875064442739328/t17179869256(17179869256) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201389.265108:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902345830a80 x1875064442739328/t17179869256(17179869256) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201389.265121:0:17857:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201389.265135:0:17857:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.265146:0:17857:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1064 (0 0) 00000080:00200000:1.0:1788201389.265153:0:17857:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201389.265179:0:17857:0:(namei.c:1628:ll_create_it()) VFS Op:name=t-29, dir=[0x200000bd1:0x40a:0x0](ffff902344979148), intent=open|creat 00000080:00200000:1.0:1788201389.265207:0:17857:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x428:0x0](ffff9023448e4b08) 00000080:00200000:1.0:1788201389.265340:0:17857:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x428:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201389.265352:0:17857:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x428:0x0](ffff9023448e4b08) 00000080:00010000:1.0:1788201389.265355:0:17857:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x428:0x0](ffff9023448e4b08) for lock 0x4605c7c97e1c543e 00000080:00010000:1.0:1788201389.265366:0:17857:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x428:0x0](ffff9023448e4b08) 00000080:00200008:1.0:1788201389.265384:0:17857:0:(file.c:1090:ll_file_open()) START file t-29:[0x200000bd1:0x428:0x0](ffff9023448e4b08), flags 100102 00000002:00100000:1.0:1788201389.265404:0:17857:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902345830a80 x1875064442739328/t17179869256(17179869256) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201389.265438:0:17857:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c543e from it ffff9023450e7060 00010000:00010000:1.0:1788201389.265442:0:17857:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.265448:0:17857:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.265465:0:17857:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x428:0x0](ffff9023448e4b08), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201389.265495:0:17857:0:(file.c:479:ll_file_release()) START file t-29:[0x200000bd1:0x428:0x0](ffff9023448e4b08), flags 100002 00000002:00100000:1.0:1788201389.265527:0:17857:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff902345830a80 x1875064442739328/t17179869256(17179869256) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201389.265534:0:17857:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201389.265538:0:17857:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902345832a00 x1875064442739584/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201389.265558:0:17857:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345832a00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442739584:192.168.201.130@tcp:35:createmany.0 00000100:00100000:1.0:1788201389.265621:0:17857:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358723f00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.276029:0:17857:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345832a00 pname:cluuid:pid:xid:nid:opc:job createmany:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17857:1875064442739584:192.168.201.130@tcp:35:createmany.0 00000002:00100000:1.0:1788201389.276066:0:17857:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff902345830a80 x1875064442739328/t17179869256(17179869256) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/664 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201389.293643:0:37:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x408:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201389.293658:0:37:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201389.293714:0:37:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x408:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.293737:0:37:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x409:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201389.293742:0:37:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201389.293761:0:37:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x409:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.293782:0:37:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x407:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201389.293786:0:37:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201389.293806:0:37:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x407:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.294921:0:17674:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.294948:0:17674:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.294970:0:17674:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:0.0:1788201389.294986:0:17674:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:0.0:1788201389.294995:0:17674:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.295002:0:17674:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.295005:0:17674:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.295013:0:17674:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181673bf8 00010000:00010000:0.0:1788201389.295017:0:17674:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:0.0:1788201389.295028:0:17674:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:0.0:1788201389.295046:0:17674:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.295055:0:17674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=267 00000080:00200000:0.0:1788201389.295062:0:17674:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 44 00000080:00200008:0.0:1788201389.295070:0:17674:0:(file.c:1090:ll_file_open()) START file d103.replay-single:[0x200000bd1:0x40a:0x0](ffff902344979148), flags 304000 00000002:00010000:0.0:1788201389.295084:0:17674:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x40a:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240304001 00010000:00010000:0.0:1788201389.295133:0:17674:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5445 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.295142:0:17674:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5445 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17674 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201389.295166:0:17674:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201389.295180:0:17674:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450b9f80 x1875064442739840/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201389.295192:0:17674:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5445 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17674 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201389.295214:0:17674:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job bash:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17674:1875064442739840:192.168.201.130@tcp:101:bash.0 00000100:00100000:0.0:1788201389.295401:0:17674:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.298667:0:17674:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job bash:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17674:1875064442739840:192.168.201.130@tcp:101:bash.0 00010000:00010000:0.0:1788201389.298693:0:17674:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5445 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.298707:0:17674:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5445 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 17674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.298715:0:17674:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5445 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 17674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.298722:0:17674:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5445 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 17674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.298732:0:17674:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5445 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 17674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.298739:0:17674:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902343731e00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.298744:0:17674:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5445 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 17674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.298751:0:17674:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5445 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 17674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.298759:0:17674:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5445 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 17674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.298766:0:17674:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.298773:0:17674:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5445 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 17674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.298780:0:17674:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731e00/0x4605c7c97e1c5445 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 17674 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201389.298800:0:17674:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9023450b9f80 x1875064442739840/t17179869258(17179869258) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'bash.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201389.298815:0:17674:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023450b9f80 x1875064442739840/t17179869258(17179869258) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'bash.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201389.298828:0:17674:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201389.298847:0:17674:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x40a:0x0], updating i_size 512 00000002:00100000:0.0:1788201389.298900:0:17674:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9023450b9f80 x1875064442739840/t17179869258(17179869258) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201405 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'bash.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788201389.298981:0:17674:0:(file.c:6283:ll_getattr_dentry()) START file d103.replay-single:[0x200000bd1:0x40a:0x0](ffff902344979148), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201389.298988:0:17674:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148),name=d103.replay-single 00000002:00010000:0.0:1788201389.299007:0:17674:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x40a:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.299019:0:17674:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 30 res 8589937617/1034 (0 0) 00010000:00010000:0.0:1788201389.299061:0:17674:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.299069:0:17674:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.299079:0:17674:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17674 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201389.299103:0:17674:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job bash:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17674:1875064442739968:192.168.201.130@tcp:101:bash.0 00000100:00100000:0.0:1788201389.299188:0:17674:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.302504:0:17674:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job bash:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17674:1875064442739968:192.168.201.130@tcp:101:bash.0 00010000:00010000:0.0:1788201389.302531:0:17674:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.302549:0:17674:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.302558:0:17674:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.302569:0:17674:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.302581:0:17674:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.302594:0:17674:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.302603:0:17674:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.302612:0:17674:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201389.302630:0:17674:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023450b8000 x1875064442739968/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/584 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201389.302643:0:17674:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.302654:0:17674:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1034 (0 0) 00000080:00200000:0.0:1788201389.302661:0:17674:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201389.302686:0:17674:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x40a:0x0], updating i_size 512 00000080:00010000:0.0:1788201389.302692:0:17674:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x40a:0x0](ffff902344979148) for lock 0x4605c7c97e1c544c 00000080:00010000:0.0:1788201389.302700:0:17674:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x40a:0x0](ffff902344979148) 00000080:00010000:0.0:1788201389.302703:0:17674:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c544c from it ffffa0b181673cd8 00010000:00010000:0.0:1788201389.302707:0:17674:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.302716:0:17674:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201389.303102:0:17674:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148) pos/size0/512 32bit_api 0 00000002:00010000:0.0:1788201389.303123:0:17674:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x40a:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: readdir flags 0 00010000:00010000:0.0:1788201389.303133:0:17674:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.303144:0:17674:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788201389.303743:0:17674:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job bash:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17674:1875064442740096:192.168.201.130@tcp:37:bash.0 00000100:00100000:0.0:1788201389.303854:0:17674:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.313851:0:17674:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job bash:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17674:1875064442740096:192.168.201.130@tcp:37:bash.0 00010000:00010000:0.0:1788201389.314696:0:17674:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.314724:0:17674:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.314803:0:17674:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148) pos/size18446744073709551614/512 32bit_api 0 00000080:00200008:0.0:1788201389.314820:0:17674:0:(file.c:479:ll_file_release()) START file d103.replay-single:[0x200000bd1:0x40a:0x0](ffff902344979148), flags 304000 00000002:00100000:0.0:1788201389.315482:0:17674:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9023450b9f80 x1875064442739840/t17179869258(17179869258) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201405 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'bash.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201389.315497:0:17674:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201389.315504:0:17674:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450b8000 x1875064442740352/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201389.315530:0:17674:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job bash:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17674:1875064442740352:192.168.201.130@tcp:35:bash.0 00000100:00100000:0.0:1788201389.315615:0:17674:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.318624:0:17674:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job bash:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17674:1875064442740352:192.168.201.130@tcp:35:bash.0 00000002:00100000:0.0:1788201389.318657:0:17674:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9023450b9f80 x1875064442739840/t17179869258(17179869258) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 648/608 e 0 to 0 dl 1788201405 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'bash.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201389.325771:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201389.325802:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.325820:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:2.0:1788201389.325837:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:2.0:1788201389.325845:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.325853:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201389.325856:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201389.325872:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:2.0:1788201389.325991:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:2.0:1788201389.326002:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:2.0:1788201389.326019:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.326027:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:2.0:1788201389.326033:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.326037:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-0, flags=8 00000080:00200008:2.0:1788201389.326043:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-0:[0x200000bd1:0x40b:0x0](ffff90234497f448), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201389.326048:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x40b:0x0](ffff90234497f448),name=t-0 00000002:00010000:2.0:1788201389.326061:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x40b:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.326071:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201389.326081:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201389.326089:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.326093:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x40b:0x0](ffff90234497f448) 00000080:00010000:2.0:1788201389.326095:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x40b:0x0](ffff90234497f448) for lock 0x4605c7c97e1c5373 00000080:00010000:2.0:1788201389.326100:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5373 from it ffffa0b181c3bcb8 00010000:00010000:2.0:1788201389.326103:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201389.326112:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201389.326227:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x40b:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201389.326239:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x40b:0x0] 00000008:00010000:2.0:1788201389.326279:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201389.326284:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398630, osc lock ffff9023500dce40, flags 0x400001002 00010000:00010000:2.0:1788201389.326337:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.326349:0:17859:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201389.326422:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442740480:192.168.201.130@tcp:101:rm.0 00000020:00200000:2.0:1788201389.326530:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788201389.331668:0:2361:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201389.331700:0:2361:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5771 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201389.331716:0:2361:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5771 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201389.331730:0:2361:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5771 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1788201389.331746:0:2361:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5771 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788201389.331758:0:2361:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1788201389.331777:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5771 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201389.331786:0:2361:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5771 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201389.331794:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5771 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201389.331803:0:2361:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5771 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1788201389.331825:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442740480:192.168.201.130@tcp:101:rm.0 00000080:00200000:2.0:1788201389.332238:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x40b:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201389.332245:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201389.332249:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.332254:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398630 (ffff902350b9de18/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.332258:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201389.332261:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398630 (ffff902350b9d7e8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201389.332306:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x40b:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201389.332491:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201389.332511:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.332528:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:2.0:1788201389.332543:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:2.0:1788201389.332551:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.332559:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201389.332563:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201389.332569:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:2.0:1788201389.332573:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:2.0:1788201389.332581:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:2.0:1788201389.332593:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.332599:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:2.0:1788201389.332604:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.332611:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-0, flags=0 00000080:00200000:2.0:1788201389.332614:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:2.0:1788201389.332619:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-0, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:2.0:1788201389.332669:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201389.332690:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c536c lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5334 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201389.332765:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201389.332788:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x40b:0x0]: losing layout lock 00010000:00010000:2.0:1788201389.332832:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201389.332841:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201389.332846:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201389.332850:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c5373 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x40b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5350 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201389.332868:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731a00/0x4605c7c97e1c544c lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d576a expref: -99 pid: 17674 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201389.332904:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201389.332916:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234518e300 x1875064442740736/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201389.332941:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518e300 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442740736:192.168.201.130@tcp:36:rm.0 00000100:00100000:2.0:1788201389.333036:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.340428:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518e300 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442740736:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201389.340516:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x40b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201389.340529:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201389.340580:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x40b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.340596:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x40b:0x0](ffff90234497f448) 00000008:00010000:0.0:1788201389.340634:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x462:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5771 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201389.340732:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.340751:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.340765:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:0.0:1788201389.340778:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:0.0:1788201389.340786:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.340792:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.340796:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.340804:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:0.0:1788201389.340807:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:0.0:1788201389.340816:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:0.0:1788201389.340831:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.340838:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.340843:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.340847:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-1, flags=8 00000080:00200008:0.0:1788201389.340855:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-1:[0x200000bd1:0x40c:0x0](ffff9023449b5348), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201389.341005:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x40c:0x0](ffff9023449b5348),name=t-1 00000002:00010000:0.0:1788201389.341021:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x40c:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.341030:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.341040:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201389.341046:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.341051:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x40c:0x0](ffff9023449b5348) 00000080:00010000:0.0:1788201389.341054:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x40c:0x0](ffff9023449b5348) for lock 0x4605c7c97e1c537a 00000080:00010000:0.0:1788201389.341058:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c537a from it ffffa0b181c3bcb8 00010000:00010000:0.0:1788201389.341061:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.341068:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.341093:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x40c:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.341099:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x40c:0x0] 00000008:00010000:0.0:1788201389.341117:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201389.341121:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff90235306d130, osc lock ffff902343fb7a20, flags 0x400001002 00010000:00010000:0.0:1788201389.341346:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.341356:0:17859:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 3/1,0 mode: --/PR res: [0x280000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788201389.341406:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1788201389.341457:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442740864:192.168.201.130@tcp:101:rm.0 00010000:00010000:1.0:1788201389.344527:0:2359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.344555:0:2359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 6/2,0 mode: --/PR res: [0x280000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5794 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.344570:0:2359:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5794 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.344583:0:2359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5794 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201389.344600:0:2359:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5794 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788201389.344610:0:2359:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1788201389.344626:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5794 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.344637:0:2359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5794 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.344645:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5794 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.344654:0:2359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5794 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201389.344661:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x40c:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201389.344670:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346370000 (ffff902348a2f128/0) at cl_lock_release():245 00000020:00010000:2.0:1788201389.344675:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346370000 (ffff902348a2f128/0) at cl_lock_cancel():142 00000100:00100000:1.0:1788201389.344677:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442740864:192.168.201.130@tcp:101:rm.0 00000020:00010000:2.0:1788201389.344681:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235306d130 (ffff902348a2fa28/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.344688:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346370000 (ffff902348a2f128/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201389.344691:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235306d130 (ffff902348a2f128/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201389.344732:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x40c:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201389.344782:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201389.344802:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.344816:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:2.0:1788201389.344829:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:2.0:1788201389.344837:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.344844:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201389.344848:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201389.344853:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:2.0:1788201389.344857:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:2.0:1788201389.344864:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:2.0:1788201389.344913:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.344920:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:2.0:1788201389.344926:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.344933:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-1, flags=0 00000080:00200000:2.0:1788201389.344936:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:2.0:1788201389.344940:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-1, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:2.0:1788201389.345010:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201389.345042:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x40c:0x0]: losing layout lock 00010000:00010000:2.0:1788201389.345328:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201389.345336:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201389.345339:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107600/0x4605c7c97e1c537a lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x40c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5373 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201389.345362:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201389.345372:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234518f100 x1875064442741120/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201389.345518:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518f100 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442741120:192.168.201.130@tcp:36:rm.0 00000100:00100000:2.0:1788201389.345605:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201389.353343:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518f100 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442741120:192.168.201.130@tcp:36:rm.0 00000080:00200000:2.0:1788201389.353441:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x40c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201389.353457:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201389.353515:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x40c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201389.353532:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x40c:0x0](ffff9023449b5348) 00000008:00010000:2.0:1788201389.353575:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x462:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5794 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201389.353665:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201389.353682:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.353697:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:2.0:1788201389.353709:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:2.0:1788201389.353717:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.353723:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201389.353726:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201389.353734:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:2.0:1788201389.353737:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:2.0:1788201389.353746:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:2.0:1788201389.353761:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.353767:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:2.0:1788201389.353772:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.353776:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-10, flags=8 00000080:00200008:2.0:1788201389.353783:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-10:[0x200000bd1:0x415:0x0](ffff9023449a4b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201389.353793:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x415:0x0](ffff9023449a4b08),name=t-10 00000002:00010000:2.0:1788201389.353819:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x415:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.353828:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201389.353836:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201389.353842:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.353846:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x415:0x0](ffff9023449a4b08) 00000080:00010000:2.0:1788201389.353848:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x415:0x0](ffff9023449a4b08) for lock 0x4605c7c97e1c53b9 00000080:00010000:2.0:1788201389.353851:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53b9 from it ffffa0b181c3bcb8 00010000:00010000:2.0:1788201389.353854:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201389.353861:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201389.354962:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x415:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201389.354976:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x415:0x0] 00000008:00010000:2.0:1788201389.355004:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201389.355008:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398630, osc lock ffff9023500dce40, flags 0x400001002 00010000:00010000:2.0:1788201389.355076:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.355088:0:17859:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201389.355132:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1788201389.355452:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442741248:192.168.201.130@tcp:101:rm.0 00010000:00010000:2.0:1788201389.364720:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.364741:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d57b7 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.364756:0:2362:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d57b7 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.364766:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d57b7 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201389.364776:0:2362:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d57b7 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201389.364784:0:2362:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788201389.364829:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d57b7 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.364841:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d57b7 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.364847:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d57b7 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.364852:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d57b7 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201389.364867:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442741248:192.168.201.130@tcp:101:rm.0 00000080:00200000:0.0:1788201389.365349:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x415:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201389.365356:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.365360:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.365365:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398630 (ffff902350b9de18/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.365369:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc5970 (ffff902350b9d7e8/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.365372:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398630 (ffff902350b9d7e8/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.365408:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x415:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.365471:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.365492:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.365945:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:0.0:1788201389.366094:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:0.0:1788201389.366104:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.366112:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.366116:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.366122:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:0.0:1788201389.366126:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:0.0:1788201389.366134:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:0.0:1788201389.366149:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.366158:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.366164:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.366172:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-10, flags=0 00000080:00200000:0.0:1788201389.366177:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.366182:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-10, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:0.0:1788201389.366244:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.366268:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x415:0x0]: losing layout lock 00010000:00010000:0.0:1788201389.366325:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.366333:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201389.366335:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802400/0x4605c7c97e1c53b9 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x415:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d54ae expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201389.366362:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201389.366373:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450b9f80 x1875064442741504/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201389.366395:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442741504:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201389.366499:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.372357:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442741504:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201389.372458:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x415:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201389.372470:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201389.372525:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x415:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.372539:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x415:0x0](ffff9023449a4b08) 00000008:00010000:0.0:1788201389.372578:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x467:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d57b7 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201389.372671:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.372688:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.372701:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:0.0:1788201389.372715:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:0.0:1788201389.372723:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.372729:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.372733:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.372740:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:0.0:1788201389.372744:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:0.0:1788201389.372753:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:0.0:1788201389.372768:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.372774:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.372780:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.372783:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-11, flags=8 00000080:00200008:0.0:1788201389.372791:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-11:[0x200000bd1:0x416:0x0](ffff9023449a21c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201389.372800:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x416:0x0](ffff9023449a21c8),name=t-11 00000002:00010000:0.0:1788201389.372810:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x416:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.372818:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.372828:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201389.372847:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.372851:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x416:0x0](ffff9023449a21c8) 00000080:00010000:0.0:1788201389.372854:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x416:0x0](ffff9023449a21c8) for lock 0x4605c7c97e1c53c0 00000080:00010000:0.0:1788201389.372858:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53c0 from it ffffa0b181c3bcb8 00010000:00010000:0.0:1788201389.372860:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.372868:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.372934:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x416:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.372941:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x416:0x0] 00000008:00010000:0.0:1788201389.373396:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201389.373400:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff90235306d630, osc lock ffff902343fb7db0, flags 0x400001002 00010000:00010000:0.0:1788201389.373451:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.373461:0:17859:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788201389.373490:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1788201389.373778:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442741632:192.168.201.130@tcp:101:rm.0 00010000:00010000:1.0:1788201389.388173:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.388202:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d57da expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.388289:0:2360:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d57da expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.388301:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d57da expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201389.388321:0:2360:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d57da expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788201389.388331:0:2360:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1788201389.388448:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d57da expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.388458:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d57da expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.388466:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d57da expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.388475:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d57da expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201389.388497:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442741632:192.168.201.130@tcp:101:rm.0 00000080:00200000:0.0:1788201389.393737:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x416:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201389.393749:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.393754:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.393760:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235306d630 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.393765:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.393768:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235306d630 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.393811:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x416:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.393864:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.393901:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.393919:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:0.0:1788201389.393936:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:0.0:1788201389.393943:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.393950:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.393953:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.393960:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:0.0:1788201389.393964:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:0.0:1788201389.393974:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:0.0:1788201389.393989:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.393996:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.394001:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.394007:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-11, flags=0 00000080:00200000:0.0:1788201389.394011:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.394016:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-11, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:0.0:1788201389.394067:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.394088:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x416:0x0]: losing layout lock 00010000:00010000:0.0:1788201389.394135:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.394141:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201389.394144:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802800/0x4605c7c97e1c53c0 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x416:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d54d1 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201389.394172:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201389.394199:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450b8380 x1875064442741888/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201389.394852:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442741888:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201389.394980:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.408214:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442741888:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201389.408314:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x416:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201389.408327:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201389.408390:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x416:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.408403:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x416:0x0](ffff9023449a21c8) 00000008:00010000:0.0:1788201389.408441:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x467:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d57da expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201389.408528:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.408545:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.408559:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:0.0:1788201389.408572:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:0.0:1788201389.408579:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.408585:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.408588:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.408596:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:0.0:1788201389.408599:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:0.0:1788201389.408607:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200000:0.0:1788201389.408623:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.408629:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.408634:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.408650:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-12, flags=8 00000080:00200008:0.0:1788201389.408659:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-12:[0x200000bd1:0x417:0x0](ffff9023449a3248), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201389.408668:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x417:0x0](ffff9023449a3248),name=t-12 00000002:00010000:0.0:1788201389.408680:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x417:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.408688:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.408696:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201389.408703:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.408707:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x417:0x0](ffff9023449a3248) 00000080:00010000:0.0:1788201389.408710:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x417:0x0](ffff9023449a3248) for lock 0x4605c7c97e1c53c7 00000080:00010000:0.0:1788201389.408714:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53c7 from it ffffa0b181c3bcb8 00010000:00010000:0.0:1788201389.408717:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.408725:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.408747:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x417:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.408754:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x417:0x0] 00000008:00010000:0.0:1788201389.408772:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201389.408781:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff90235306d630, osc lock ffff902343fb7db0, flags 0x400001002 00010000:00010000:0.0:1788201389.408829:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 3/1,0 mode: --/PR res: [0x240000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.408838:0:17859:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 3/1,0 mode: --/PR res: [0x240000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788201389.408924:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1788201389.409324:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442742016:192.168.201.130@tcp:101:rm.0 00010000:00010000:1.0:1788201389.412454:0:2359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.412483:0:2359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 6/2,0 mode: --/PR res: [0x240000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d57fd expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.412504:0:2359:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d57fd expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.412517:0:2359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d57fd expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201389.412531:0:2359:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d57fd expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788201389.412544:0:2359:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1788201389.412561:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d57fd expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.412570:0:2359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d57fd expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.412577:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d57fd expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.412585:0:2359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d57fd expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201389.412605:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442742016:192.168.201.130@tcp:101:rm.0 00000080:00200000:0.0:1788201389.412754:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x417:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201389.412760:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.412763:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.412768:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235306d630 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.412772:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.412774:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235306d630 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.412812:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x417:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.412864:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.412914:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.412929:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:0.0:1788201389.412942:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200000:0.0:1788201389.412950:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.412993:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.412997:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.413003:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:0.0:1788201389.413006:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:0.0:1788201389.413013:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00200000:0.0:1788201389.413032:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.413040:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.413045:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.413052:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-12, flags=0 00000080:00200000:0.0:1788201389.413057:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.413063:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-12, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:0.0:1788201389.413115:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.413138:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x417:0x0]: losing layout lock 00010000:00010000:0.0:1788201389.413181:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.413194:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201389.413197:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803a00/0x4605c7c97e1c53c7 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x417:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d54f4 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201389.413298:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201389.413309:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450b9c00 x1875064442742272/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201389.413366:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442742272:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201389.413463:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe0b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.421360:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442742272:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201389.421445:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x417:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201389.421456:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201389.421663:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x417:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.421677:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x417:0x0](ffff9023449a3248) 00000008:00010000:0.0:1788201389.421719:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x468:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d57fd expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201389.421815:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.421831:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.421844:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:0.0:1788201389.421856:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00200000:0.0:1788201389.421868:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.421874:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.421986:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.421993:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:0.0:1788201389.421998:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:0.0:1788201389.422007:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000080:00200000:0.0:1788201389.422022:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.422028:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.422033:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.422037:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-13, flags=8 00000080:00200008:0.0:1788201389.422044:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-13:[0x200000bd1:0x418:0x0](ffff9023449a42c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201389.422053:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x418:0x0](ffff9023449a42c8),name=t-13 00000002:00010000:1.0:1788201389.422194:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x418:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.422216:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.422228:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.422235:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.422241:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x418:0x0](ffff9023449a42c8) 00000080:00010000:1.0:1788201389.422244:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x418:0x0](ffff9023449a42c8) for lock 0x4605c7c97e1c53ce 00000080:00010000:1.0:1788201389.422249:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53ce from it ffffa0b181c3bcb8 00010000:00010000:1.0:1788201389.422252:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.422260:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201389.422302:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x418:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788201389.422310:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x418:0x0] 00000008:00010000:1.0:1788201389.422337:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788201389.422341:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023450455b0, osc lock ffff90235090c000, flags 0x400001002 00010000:00010000:1.0:1788201389.422389:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.422399:0:17859:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788201389.422437:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1788201389.422447:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201389.422463:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345831180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442742400:192.168.201.130@tcp:101:rm.0 00010000:00010000:1.0:1788201389.427728:0:2359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.427770:0:2359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5820 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.427787:0:2359:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5820 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.427800:0:2359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5820 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201389.427817:0:2359:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5820 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788201389.427828:0:2359:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1788201389.427846:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5820 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.427855:0:2359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5820 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.427863:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5820 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.427915:0:2359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5820 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201389.427937:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345831180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442742400:192.168.201.130@tcp:101:rm.0 00000080:00200000:2.0:1788201389.428092:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x418:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201389.428099:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235844f090 (ffff9023500e7098/0) at cl_lock_release():245 00000020:00010000:2.0:1788201389.428102:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235844f090 (ffff9023500e7098/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.428107:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023450455b0 (ffff9023500e7878/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.428111:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235844f090 (ffff9023500e7098/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201389.428113:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023450455b0 (ffff9023500e7098/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201389.428150:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x418:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201389.428386:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201389.428409:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.428425:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:2.0:1788201389.428437:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000080:00200000:2.0:1788201389.428445:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.428452:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201389.428456:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201389.428461:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:2.0:1788201389.428465:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:2.0:1788201389.428472:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000080:00200000:2.0:1788201389.428483:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.428489:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:2.0:1788201389.428495:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.428501:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-13, flags=0 00000080:00200000:2.0:1788201389.428506:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:2.0:1788201389.428510:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-13, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:2.0:1788201389.428745:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201389.428836:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x418:0x0]: losing layout lock 00010000:00010000:2.0:1788201389.429119:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201389.429289:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201389.429293:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802c00/0x4605c7c97e1c53ce lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x418:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5517 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201389.429320:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201389.429331:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234518c380 x1875064442742656/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201389.429354:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518c380 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442742656:192.168.201.130@tcp:36:rm.0 00000100:00100000:2.0:1788201389.429503:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201389.440444:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518c380 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442742656:192.168.201.130@tcp:36:rm.0 00000080:00200000:2.0:1788201389.440538:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x418:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201389.440558:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201389.440609:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x418:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201389.440622:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x418:0x0](ffff9023449a42c8) 00000008:00010000:2.0:1788201389.440660:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x468:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5820 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201389.440747:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201389.440774:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.440787:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:2.0:1788201389.440801:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000080:00200000:2.0:1788201389.440809:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.440815:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201389.440818:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201389.440825:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:2.0:1788201389.440829:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:2.0:1788201389.440837:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00000080:00200000:2.0:1788201389.440851:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.440857:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:2.0:1788201389.440862:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.440866:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-14, flags=8 00000080:00200008:2.0:1788201389.440873:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-14:[0x200000bd1:0x419:0x0](ffff9023449a3a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201389.443055:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x419:0x0](ffff9023449a3a88),name=t-14 00000002:00010000:1.0:1788201389.443095:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x419:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.443116:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.443130:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.443138:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.443144:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x419:0x0](ffff9023449a3a88) 00000080:00010000:1.0:1788201389.443147:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x419:0x0](ffff9023449a3a88) for lock 0x4605c7c97e1c53d5 00000080:00010000:1.0:1788201389.443154:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53d5 from it ffffa0b181c3bcb8 00010000:00010000:1.0:1788201389.443172:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.443181:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201389.443232:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x419:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788201389.443241:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x419:0x0] 00000008:00010000:1.0:1788201389.443296:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788201389.443300:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902345045db0, osc lock ffff90235090d7c0, flags 0x400001002 00010000:00010000:1.0:1788201389.443353:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 3/1,0 mode: --/PR res: [0x240000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.443364:0:17859:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 3/1,0 mode: --/PR res: [0x240000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788201389.443399:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1788201389.443422:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345831180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442742784:192.168.201.130@tcp:101:rm.0 00010000:00010000:0.0:1788201389.443777:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.443785:0:17860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201389.443790:0:17860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53dc from it ffffa0b181df7d78 00010000:00010000:0.0:1788201389.443792:0:17860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.443797:0:17860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201389.444479:0:17861:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41a:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201389.444603:0:17861:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x41a:0x0] 00000008:00010000:2.0:1788201389.444638:0:17861:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:2.0:1788201389.444642:0:17861:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398a30, osc lock ffff9023500dd8f0, flags 0x400041012 00010000:00010000:2.0:1788201389.444683:0:17861:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.444693:0:17861:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1788201389.444728:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201389.444732:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.444735:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398a30 (ffff902350b9df38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.444738:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201389.444741:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398a30 (ffff902350b9d1b8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201389.444767:0:17861:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41a:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1788201389.444785:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201389.444799:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442742912:192.168.201.130@tcp:101:ll_agl.0 00010000:00010000:0.0:1788201389.445484:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.445499:0:17860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201389.445507:0:17860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53e3 from it ffffa0b181df7d78 00010000:00010000:0.0:1788201389.445511:0:17860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.445518:0:17860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201389.445965:0:17861:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41b:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201389.445973:0:17861:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x41b:0x0] 00000008:00010000:2.0:1788201389.445993:0:17861:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:2.0:1788201389.445996:0:17861:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398a30, osc lock ffff9023500dd8f0, flags 0x400041012 00010000:00010000:2.0:1788201389.446040:0:17861:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 3/1,0 mode: --/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.446050:0:17861:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 3/1,0 mode: --/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.446361:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.446372:0:17860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201389.446380:0:17860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53ea from it ffffa0b181df7d78 00010000:00010000:0.0:1788201389.446383:0:17860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.446391:0:17860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.446413:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.446423:0:17860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201389.446429:0:17860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53f1 from it ffffa0b181df7d78 00010000:00010000:0.0:1788201389.446432:0:17860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.446438:0:17860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.446459:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.446466:0:17860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201389.446472:0:17860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53f8 from it ffffa0b181df7d78 00010000:00010000:0.0:1788201389.446475:0:17860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.446482:0:17860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.446500:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.446508:0:17860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201389.446514:0:17860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53ff from it ffffa0b181df7d78 00010000:00010000:0.0:1788201389.446517:0:17860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.446523:0:17860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00010000:2.0:1788201389.448036:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201389.448043:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.448048:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398a30 (ffff902350b9df38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.448052:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201389.448055:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398a30 (ffff902350b9d1b8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201389.448463:0:17861:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41b:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1788201389.448507:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442743040:192.168.201.130@tcp:101:ll_agl.0 00010000:00010000:0.0:1788201389.448728:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.448740:0:17860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201389.448747:0:17860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5406 from it ffffa0b181df7d78 00010000:00010000:0.0:1788201389.449294:0:17860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.449306:0:17860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.449335:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.449343:0:17860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201389.449350:0:17860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c540d from it ffffa0b181df7d78 00010000:00010000:0.0:1788201389.449354:0:17860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.449360:0:17860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.449398:0:17860:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41c:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.449406:0:17860:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x41c:0x0] 00000008:00010000:0.0:1788201389.449430:0:17860:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:0.0:1788201389.449433:0:17860:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff90235306d730, osc lock ffff902343fb7db0, flags 0x400041012 00010000:00010000:0.0:1788201389.449480:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.449490:0:17860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.449575:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442743168:192.168.201.130@tcp:101:ll_sa.0 00000080:00200000:2.0:1788201389.449777:0:17861:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201389.449783:0:17861:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x41d:0x0] 00000008:00010000:2.0:1788201389.449804:0:17861:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:2.0:1788201389.449806:0:17861:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398a30, osc lock ffff9023500dd8f0, flags 0x400041012 00010000:00010000:2.0:1788201389.449833:0:17861:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.449841:0:17861:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1788201389.449861:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201389.449864:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.449867:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398a30 (ffff902350b9df38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.449870:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201389.449872:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398a30 (ffff902350b9d1b8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201389.449910:0:17861:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41d:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1788201389.449928:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201389.449936:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442743296:192.168.201.130@tcp:101:ll_agl.0 00000020:00010000:0.0:1788201389.450059:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.450071:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.450074:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235306d730 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.450076:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.450078:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235306d730 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.450098:0:17860:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41c:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.450118:0:17860:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.450123:0:17860:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x41e:0x0] 00000008:00010000:0.0:1788201389.450135:0:17860:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:0.0:1788201389.450137:0:17860:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff90235306d730, osc lock ffff902343fb7db0, flags 0x400041012 00010000:00010000:0.0:1788201389.450174:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.450187:0:17860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:0.0:1788201389.450200:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.450203:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.450205:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235306d730 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.450208:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.450210:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235306d730 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.450222:0:17860:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.450237:0:17860:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41f:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.450242:0:17860:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x41f:0x0] 00000008:00010000:0.0:1788201389.450253:0:17860:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:0.0:1788201389.450254:0:17860:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff90235306d730, osc lock ffff902343fb7db0, flags 0x400041012 00010000:00010000:0.0:1788201389.450286:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.450294:0:17860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:0.0:1788201389.450305:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.450308:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.450311:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235306d730 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.450313:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.450316:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235306d730 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.450329:0:17860:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41f:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.450350:0:17860:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x420:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.450355:0:17860:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x420:0x0] 00000008:00010000:0.0:1788201389.450366:0:17860:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:0.0:1788201389.450368:0:17860:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff90235306d730, osc lock ffff902343fb7db0, flags 0x400041012 00010000:00010000:0.0:1788201389.450387:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 3/1,0 mode: --/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.450395:0:17860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 3/1,0 mode: --/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:0.0:1788201389.450404:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.450406:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.450409:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235306d730 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.450411:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.450413:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235306d730 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.450426:0:17860:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x420:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.450443:0:17860:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x421:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.450447:0:17860:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x421:0x0] 00000008:00010000:0.0:1788201389.450458:0:17860:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:0.0:1788201389.450460:0:17860:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff90235306d730, osc lock ffff902343fb7db0, flags 0x400041012 00010000:00010000:0.0:1788201389.450479:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.450487:0:17860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:0.0:1788201389.450496:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.450499:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.450501:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235306d730 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.450504:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.450506:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235306d730 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.450519:0:17860:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x421:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:0.0:1788201389.450576:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442743424:192.168.201.130@tcp:101:ll_sa.0 00000100:00100000:1.0:1788201389.451289:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442743552:192.168.201.130@tcp:101:ll_sa.0 00000100:00100000:1.0:1788201389.451322:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442743808:192.168.201.130@tcp:101:ll_sa.0 00000100:00100000:0.0:1788201389.451976:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442743680:192.168.201.130@tcp:101:ll_sa.0 00010000:00010000:0.0:1788201389.452193:0:2359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.452208:0:2359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 6/2,0 mode: --/PR res: [0x240000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5843 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.452220:0:2359:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5843 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.452230:0:2359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5843 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788201389.452240:0:2359:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5843 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788201389.452249:0:2359:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1788201389.452262:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5843 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.452286:0:2359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5843 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.452293:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5843 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.452301:0:2359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5843 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201389.452305:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x419:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201389.452311:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235844f2e0 (ffff9023500e72d8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201389.452314:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235844f2e0 (ffff9023500e72d8/0) at cl_lock_cancel():142 00000100:00100000:0.0:1788201389.452316:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345831180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442742784:192.168.201.130@tcp:101:rm.0 00000020:00010000:2.0:1788201389.452318:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902345045db0 (ffff9023500e77e8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.452322:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235844f2e0 (ffff9023500e72d8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201389.452324:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902345045db0 (ffff9023500e72d8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201389.452427:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x419:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201389.452487:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201389.453156:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.453173:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00010000:00010000:2.0:1788201389.453186:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00000080:00200000:2.0:1788201389.453194:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.453203:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201389.453206:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201389.453212:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:2.0:1788201389.453216:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00010000:00010000:2.0:1788201389.453223:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00000080:00200000:2.0:1788201389.453235:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.453242:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:2.0:1788201389.453248:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.453256:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-14, flags=0 00000080:00200000:2.0:1788201389.453285:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:2.0:1788201389.453292:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-14, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:2.0:1788201389.453413:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201389.453436:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x419:0x0]: losing layout lock 00010000:00010000:2.0:1788201389.453474:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201389.453478:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201389.453481:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803c00/0x4605c7c97e1c53d5 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x419:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d553a expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201389.453686:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201389.453694:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234518ed80 x1875064442744064/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201389.453711:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518ed80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442744064:192.168.201.130@tcp:36:rm.0 00000100:00100000:2.0:1788201389.453771:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3600 going to sleep for 11 seconds 00010000:00010000:2.0:1788201389.454699:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.454718:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.454731:0:2362:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.454741:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201389.454789:0:2362:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.454815:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.454824:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.454831:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.454838:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201389.454854:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442742912:192.168.201.130@tcp:101:ll_agl.0 00010000:00010000:0.0:1788201389.466036:0:2359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.466073:0:2359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.466089:0:2359:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.466101:0:2359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788201389.466146:0:2359:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.466305:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.466315:0:2359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.466323:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.466331:0:2359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788201389.466353:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442743424:192.168.201.130@tcp:101:ll_sa.0 00010000:00010000:0.0:1788201389.466372:0:2359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 6/2,0 mode: --/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.466442:0:2359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 6/2,0 mode: --/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.466453:0:2359:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.466461:0:2359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788201389.466481:0:2359:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.466497:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.466505:0:2359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.466512:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.466521:0:2359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201389.466695:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.466708:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.466717:0:2362:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.466727:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201389.466752:0:2362:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.466772:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.466780:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.466787:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.466795:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201389.466807:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442743296:192.168.201.130@tcp:101:ll_agl.0 00010000:00010000:2.0:1788201389.466823:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.466833:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.466842:0:2362:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.466850:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201389.466868:0:2362:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.466920:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.466928:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.466936:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.466944:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201389.466955:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442743040:192.168.201.130@tcp:101:ll_agl.0 00000100:00100000:0.0:1788201389.467033:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442743680:192.168.201.130@tcp:101:ll_sa.0 00010000:00010000:1.0:1788201389.467457:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467470:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467479:0:2360:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467488:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201389.467515:0:2360:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467673:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467683:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467692:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467700:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201389.467714:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442743552:192.168.201.130@tcp:101:ll_sa.0 00010000:00010000:1.0:1788201389.467732:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467740:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467762:0:2360:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467772:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201389.467791:0:2360:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467807:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467814:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467821:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467829:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201389.467840:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442743808:192.168.201.130@tcp:101:ll_sa.0 00010000:00010000:1.0:1788201389.467855:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467865:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467907:0:2360:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467918:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201389.467936:0:2360:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467952:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467959:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467966:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.467974:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201389.467984:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442743168:192.168.201.130@tcp:101:ll_sa.0 00000100:00100000:1.0:1788201389.471671:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518ed80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442744064:192.168.201.130@tcp:36:rm.0 00000080:00200000:1.0:1788201389.471771:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x419:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201389.471784:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201389.471837:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x419:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201389.471853:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x419:0x0](ffff9023449a3a88) 00000008:00010000:1.0:1788201389.471936:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x469:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5843 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.472018:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.472036:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.472049:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00010000:00010000:1.0:1788201389.472071:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00000080:00200000:1.0:1788201389.472079:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.472085:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.472088:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.472095:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:1.0:1788201389.472099:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00010000:00010000:1.0:1788201389.472108:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00000080:00200000:1.0:1788201389.472132:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.472140:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:1.0:1788201389.472145:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.472149:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-15, flags=8 00010000:00010000:1.0:1788201389.472961:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1788201389.472980:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53dc from it ffffa0b181c3baf8 00010000:00010000:1.0:1788201389.472983:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201389.472991:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00000080:00200008:1.0:1788201389.473138:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-15:[0x200000bd1:0x41a:0x0](ffff9023449b1148), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201389.473145:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x41a:0x0](ffff9023449b1148),name=t-15 00000002:00010000:1.0:1788201389.473178:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x41a:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.473187:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201389.473198:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1788201389.473205:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.473210:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x41a:0x0](ffff9023449b1148) 00000080:00010000:1.0:1788201389.473213:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x41a:0x0](ffff9023449b1148) for lock 0x4605c7c97e1c53dc 00000080:00010000:1.0:1788201389.473218:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53dc from it ffffa0b181c3bcb8 00010000:00010000:1.0:1788201389.473222:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201389.473230:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000080:00200000:1.0:1788201389.473284:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41a:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788201389.473292:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x41a:0x0] 00000008:00010000:1.0:1788201389.473320:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788201389.473323:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902345045c30, osc lock ffff90235090cd10, flags 0x400001002 00010000:00010000:1.0:1788201389.473343:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x469:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.473354:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:1.0:1788201389.473368:0:17859:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:1.0:1788201389.473388:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788201389.473392:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.473400:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:1.0:1788201389.473411:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788201389.473423:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x41a:0x0] updating i_size 0 i_blocks 1 00000020:00010000:1.0:1788201389.473427:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235844f530 (ffff9023500e7a28/0) at cl_lock_release():245 00000020:00010000:1.0:1788201389.473430:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235844f530 (ffff9023500e7a28/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201389.473432:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902345045c30 (ffff9023500e7e18/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788201389.473437:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235844f530 (ffff9023500e7a28/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201389.473440:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902345045c30 (ffff9023500e7a28/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201389.473462:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41a:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201389.473497:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.473506:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.473513:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00010000:00010000:1.0:1788201389.473523:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00000080:00200000:1.0:1788201389.473541:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.473545:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.473555:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.473560:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:1.0:1788201389.473563:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00010000:00010000:1.0:1788201389.473571:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00000080:00200000:1.0:1788201389.473580:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.473587:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:1.0:1788201389.473591:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.473596:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-15, flags=0 00000080:00200000:1.0:1788201389.473598:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201389.473603:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-15, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:1.0:1788201389.473640:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000080:00200000:1.0:1788201389.473661:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x41a:0x0]: losing layout lock 00010000:00010000:1.0:1788201389.473705:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1788201389.473712:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201389.473715:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803000/0x4605c7c97e1c53dc lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x41a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d555d expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000020:00100000:1.0:1788201389.473741:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201389.473752:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902345830000 x1875064442744320/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201389.473789:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345830000 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442744320:192.168.201.130@tcp:36:rm.0 00000100:00100000:1.0:1788201389.473915:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358723f00 going to sleep for 11 seconds 00010000:00010000:0.0:1788201389.475678:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.475699:0:17860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201389.475707:0:17860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5414 from it ffffa0b181df7d78 00010000:00010000:0.0:1788201389.475713:0:17860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.475723:0:17860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000100:00100000:1.0:1788201389.479251:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345830000 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442744320:192.168.201.130@tcp:36:rm.0 00000080:00200000:1.0:1788201389.479345:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201389.479356:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201389.479416:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201389.479432:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x41a:0x0](ffff9023449b1148) 00000008:00010000:1.0:1788201389.479474:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x469:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201389.479575:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.479591:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.479605:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00010000:00010000:1.0:1788201389.479619:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00000080:00200000:1.0:1788201389.479627:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.479633:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.479636:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.479643:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:1.0:1788201389.479647:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00010000:00010000:1.0:1788201389.479655:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00000080:00200000:1.0:1788201389.479670:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.479678:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:1.0:1788201389.479682:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.479686:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-16, flags=8 00010000:00010000:1.0:1788201389.479695:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1788201389.479703:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53e3 from it ffffa0b181c3baf8 00010000:00010000:1.0:1788201389.479706:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201389.479713:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:1.0:1788201389.479737:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-16:[0x200000bd1:0x41b:0x0](ffff9023449b00c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201389.479742:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x41b:0x0](ffff9023449b00c8),name=t-16 00000002:00010000:1.0:1788201389.479753:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x41b:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.479759:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201389.479768:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201389.479773:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.479777:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x41b:0x0](ffff9023449b00c8) 00000080:00010000:1.0:1788201389.479779:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x41b:0x0](ffff9023449b00c8) for lock 0x4605c7c97e1c53e3 00000080:00010000:1.0:1788201389.479782:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53e3 from it ffffa0b181c3bcb8 00010000:00010000:1.0:1788201389.479785:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201389.479792:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000080:00200000:1.0:1788201389.479815:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41b:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788201389.479821:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x41b:0x0] 00000008:00010000:1.0:1788201389.479838:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788201389.479842:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902345045c30, osc lock ffff90235090cd10, flags 0x400001002 00010000:00010000:1.0:1788201389.479857:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.479871:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:1.0:1788201389.479911:0:17859:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:1.0:1788201389.479922:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788201389.479925:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.479933:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:1.0:1788201389.479942:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788201389.479951:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x41b:0x0] updating i_size 0 i_blocks 1 00000020:00010000:1.0:1788201389.479955:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235844f530 (ffff9023500e7a28/0) at cl_lock_release():245 00000020:00010000:1.0:1788201389.479958:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235844f530 (ffff9023500e7a28/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201389.479960:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902345045c30 (ffff9023500e7e18/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788201389.479965:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235844f530 (ffff9023500e7a28/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201389.479968:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902345045c30 (ffff9023500e7a28/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201389.479982:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41b:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201389.480004:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.480011:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.480017:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00010000:00010000:1.0:1788201389.480027:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00000080:00200000:1.0:1788201389.480033:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.480037:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.480040:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.480043:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:1.0:1788201389.480046:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00010000:00010000:1.0:1788201389.480053:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00000080:00200000:1.0:1788201389.480062:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.480066:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:1.0:1788201389.480069:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.480073:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-16, flags=0 00000080:00200000:1.0:1788201389.480076:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201389.480081:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-16, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:1.0:1788201389.480118:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000080:00200000:1.0:1788201389.480139:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x41b:0x0]: losing layout lock 00010000:00010000:1.0:1788201389.480181:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1788201389.480188:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201389.480192:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c53e3 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x41b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5580 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 1 00000020:00100000:1.0:1788201389.480216:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201389.480225:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902345830e00 x1875064442744576/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201389.480247:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345830e00 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442744576:192.168.201.130@tcp:36:rm.0 00000100:00100000:1.0:1788201389.480331:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358723f00 going to sleep for 11 seconds 00000080:00200000:2.0:1788201389.482017:0:17861:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x422:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201389.482029:0:17861:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x422:0x0] 00010000:00010000:0.0:1788201389.482320:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.482335:0:17860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201389.482342:0:17860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c541b from it ffffa0b181df7d78 00010000:00010000:0.0:1788201389.482348:0:17860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.482357:0:17860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1788201389.482425:0:17861:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:2.0:1788201389.482429:0:17861:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398a30, osc lock ffff9023500dd8f0, flags 0x400041012 00010000:00010000:2.0:1788201389.482475:0:17861:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 3/1,0 mode: --/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.482485:0:17861:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 3/1,0 mode: --/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201389.482564:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442744704:192.168.201.130@tcp:101:ll_agl.0 00010000:00010000:0.0:1788201389.482656:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.482665:0:17860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201389.482672:0:17860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5422 from it ffffa0b181df7d78 00010000:00010000:0.0:1788201389.482675:0:17860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.482683:0:17860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.482702:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.482710:0:17860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201389.482716:0:17860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5429 from it ffffa0b181df7d78 00010000:00010000:0.0:1788201389.482719:0:17860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.482726:0:17860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.482740:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.482748:0:17860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201389.482764:0:17860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5430 from it ffffa0b181df7d78 00010000:00010000:0.0:1788201389.482768:0:17860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.482774:0:17860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.482791:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.482799:0:17860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201389.482806:0:17860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5437 from it ffffa0b181df7d78 00010000:00010000:0.0:1788201389.482809:0:17860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.482816:0:17860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.482830:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.482837:0:17860:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1788201389.482843:0:17860:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c543e from it ffffa0b181df7d78 00010000:00010000:0.0:1788201389.482846:0:17860:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.482853:0:17860:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.482910:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6600/0x4605c7c97e1c54c3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.482917:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6600/0x4605c7c97e1c54c3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.482937:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7e00/0x4605c7c97e1c54ca lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.482943:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7e00/0x4605c7c97e1c54ca lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.482958:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6e00/0x4605c7c97e1c54d1 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.482964:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6e00/0x4605c7c97e1c54d1 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.482978:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6200/0x4605c7c97e1c54d8 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 6 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.482984:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6200/0x4605c7c97e1c54d8 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.482997:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6c00/0x4605c7c97e1c54df lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 7 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483003:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6c00/0x4605c7c97e1c54df lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483016:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7000/0x4605c7c97e1c54e6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 8 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483022:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7000/0x4605c7c97e1c54e6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483038:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7a00/0x4605c7c97e1c54ed lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 9 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483044:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7a00/0x4605c7c97e1c54ed lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483057:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6400/0x4605c7c97e1c54f4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 10 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483072:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6400/0x4605c7c97e1c54f4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483086:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7600/0x4605c7c97e1c54fb lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 11 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483093:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7600/0x4605c7c97e1c54fb lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483106:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7400/0x4605c7c97e1c5502 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 12 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483111:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7400/0x4605c7c97e1c5502 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483134:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7200/0x4605c7c97e1c5509 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 13 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483140:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7200/0x4605c7c97e1c5509 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483154:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b800/0x4605c7c97e1c5510 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 14 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483168:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b800/0x4605c7c97e1c5510 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483183:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b600/0x4605c7c97e1c5517 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 15 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483190:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b600/0x4605c7c97e1c5517 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483202:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c551e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 16 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483208:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c551e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483221:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a000/0x4605c7c97e1c5525 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 17 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483227:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a000/0x4605c7c97e1c5525 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483239:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c552c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 18 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483245:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c552c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483257:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c5533 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 19 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483263:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c5533 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 19 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483288:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c553a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 20 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483295:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c553a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483308:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5541 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 21 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483313:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5541 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1788201389.483383:0:17860:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x423:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.483390:0:17860:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x423:0x0] 00000008:00010000:0.0:1788201389.483408:0:17860:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:0.0:1788201389.483411:0:17860:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023458178b0, osc lock ffff902343fb7db0, flags 0x400041012 00010000:00010000:0.0:1788201389.483434:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483443:0:17860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:0.0:1788201389.483455:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.483457:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.483460:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023458178b0 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.483463:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.483465:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023458178b0 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.483492:0:17860:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x423:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.483508:0:17860:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x424:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.483512:0:17860:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x424:0x0] 00000008:00010000:0.0:1788201389.483521:0:17860:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:0.0:1788201389.483524:0:17860:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023458178b0, osc lock ffff902343fb7db0, flags 0x400041012 00010000:00010000:0.0:1788201389.483539:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483547:0:17860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 3/1,0 mode: --/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:0.0:1788201389.483556:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.483558:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.483560:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023458178b0 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.483562:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.483564:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023458178b0 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.483575:0:17860:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x424:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.483590:0:17860:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x425:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.483594:0:17860:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x425:0x0] 00000008:00010000:0.0:1788201389.483604:0:17860:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:0.0:1788201389.483606:0:17860:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023458178b0, osc lock ffff902343fb7db0, flags 0x400041012 00010000:00010000:0.0:1788201389.483620:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483628:0:17860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:0.0:1788201389.483636:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.483638:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.483640:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023458178b0 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.483643:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.483645:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023458178b0 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.483654:0:17860:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x425:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.483669:0:17860:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x426:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.483672:0:17860:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x426:0x0] 00000008:00010000:0.0:1788201389.483681:0:17860:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:0.0:1788201389.483683:0:17860:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023458178b0, osc lock ffff902343fb7db0, flags 0x400041012 00010000:00010000:0.0:1788201389.483696:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483704:0:17860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 3/1,0 mode: --/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:0.0:1788201389.483711:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.483713:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.483715:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023458178b0 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.483718:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.483720:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023458178b0 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.483730:0:17860:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x426:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.483744:0:17860:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x427:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.483748:0:17860:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x427:0x0] 00000008:00010000:0.0:1788201389.483757:0:17860:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:0.0:1788201389.483759:0:17860:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023458178b0, osc lock ffff902343fb7db0, flags 0x400041012 00010000:00010000:0.0:1788201389.483773:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483781:0:17860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:0.0:1788201389.483788:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.483791:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.483794:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023458178b0 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.483796:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.483798:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023458178b0 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.483808:0:17860:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x427:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.483822:0:17860:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x428:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.483826:0:17860:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x428:0x0] 00000008:00010000:0.0:1788201389.483835:0:17860:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 2b 00000008:00010000:0.0:1788201389.483836:0:17860:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023458178b0, osc lock ffff902343fb7db0, flags 0x400041012 00010000:00010000:0.0:1788201389.483852:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.483859:0:17860:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400041010 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 3/1,0 mode: --/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00010000:0.0:1788201389.483867:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.483869:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.483871:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023458178b0 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.483874:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.483897:0:17860:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023458178b0 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.483908:0:17860:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x428:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000020:00010000:2.0:1788201389.484660:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201389.484665:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.484669:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398a30 (ffff902350b9df38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.484672:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201389.484674:0:17861:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398a30 (ffff902350b9d1b8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201389.484700:0:17861:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x422:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:1.0:1788201389.484808:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442744960:192.168.201.130@tcp:101:ll_sa.0 00000100:00100000:1.0:1788201389.484873:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442745216:192.168.201.130@tcp:101:ll_sa.0 00000100:00100000:1.0:1788201389.484905:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442745472:192.168.201.130@tcp:101:ll_sa.0 00000100:00100000:1.0:1788201389.484925:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [0->1] 00000100:00100000:1.0:1788201389.484931:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442744832:192.168.201.130@tcp:63:ll_sa.0 00000100:00100000:1.0:1788201389.484990:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442745088:192.168.201.130@tcp:101:ll_sa.0 00000100:00100000:1.0:1788201389.485006:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442745344:192.168.201.130@tcp:101:ll_sa.0 00000100:00100000:1.0:1788201389.485020:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442745600:192.168.201.130@tcp:101:ll_sa.0 00010000:00010000:1.0:1788201389.490745:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.490783:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.490804:0:2360:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.490816:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201389.490865:0:2360:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.490969:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.490978:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.490985:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.490993:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201389.491013:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442744960:192.168.201.130@tcp:101:ll_sa.0 00010000:00010000:1.0:1788201389.491029:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.491041:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.491049:0:2360:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.491056:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201389.491089:0:2360:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.491106:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.491113:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.491120:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.491126:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201389.491141:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442745216:192.168.201.130@tcp:101:ll_sa.0 00010000:00010000:2.0:1788201389.491855:0:2362:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.491871:0:2362:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 6/2,0 mode: --/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.491906:0:2362:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.491916:0:2362:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201389.491954:0:2362:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.491979:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.491988:0:2362:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.491994:0:2362:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.492002:0:2362:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201389.492017:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442744704:192.168.201.130@tcp:101:ll_agl.0 00010000:00010000:1.0:1788201389.492910:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.492928:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 6/2,0 mode: --/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.492939:0:2360:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.492948:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201389.492978:0:2360:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493003:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493012:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493019:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493028:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201389.493043:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442745088:192.168.201.130@tcp:101:ll_sa.0 00010000:00010000:1.0:1788201389.493056:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493076:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 6/2,0 mode: --/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493086:0:2360:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493093:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201389.493111:0:2360:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493128:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493136:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493144:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493152:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201389.493172:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442745344:192.168.201.130@tcp:101:ll_sa.0 00010000:00010000:1.0:1788201389.493187:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493205:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 6/2,0 mode: --/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493214:0:2360:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493222:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201389.493240:0:2360:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493258:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493265:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493282:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493288:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201389.493298:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442745600:192.168.201.130@tcp:101:ll_sa.0 00010000:00010000:1.0:1788201389.493313:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493323:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493331:0:2360:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493339:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201389.493356:0:2360:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493383:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493391:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493397:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.493403:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201389.493413:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442745472:192.168.201.130@tcp:101:ll_sa.0 00000100:00100000:1.0:1788201389.496689:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345830e00 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442744576:192.168.201.130@tcp:36:rm.0 00000080:00200000:1.0:1788201389.496848:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201389.496860:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201389.497172:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201389.497195:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x41b:0x0](ffff9023449b00c8) 00000008:00010000:1.0:1788201389.497233:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201389.497323:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.497341:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.497355:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00010000:00010000:1.0:1788201389.497367:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00000080:00200000:1.0:1788201389.497384:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.497390:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.497395:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.497403:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:1.0:1788201389.497406:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00010000:00010000:1.0:1788201389.497413:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00000080:00200000:1.0:1788201389.497428:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.497435:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:1.0:1788201389.497439:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.497443:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-17, flags=8 00010000:00010000:1.0:1788201389.497453:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1788201389.497461:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53ea from it ffffa0b181c3baf8 00010000:00010000:1.0:1788201389.497464:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201389.497472:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:1.0:1788201389.497499:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-17:[0x200000bd1:0x41c:0x0](ffff9023449b3248), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201389.497505:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x41c:0x0](ffff9023449b3248),name=t-17 00000002:00010000:1.0:1788201389.497517:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x41c:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.497523:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201389.497541:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201389.497687:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.497692:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x41c:0x0](ffff9023449b3248) 00000080:00010000:1.0:1788201389.497695:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x41c:0x0](ffff9023449b3248) for lock 0x4605c7c97e1c53ea 00000080:00010000:1.0:1788201389.497699:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53ea from it ffffa0b181c3bcb8 00010000:00010000:1.0:1788201389.497702:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201389.497709:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201389.497733:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41c:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788201389.497740:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x41c:0x0] 00000008:00010000:1.0:1788201389.497992:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788201389.497997:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902345045c30, osc lock ffff90235090cd10, flags 0x400001002 00010000:00010000:1.0:1788201389.498014:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.498031:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:1.0:1788201389.498044:0:17859:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:1.0:1788201389.498054:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788201389.498194:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70c00/0x4605c7c97e1c5572 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 22 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498202:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70c00/0x4605c7c97e1c5572 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498223:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71a00/0x4605c7c97e1c5579 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 23 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498230:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71a00/0x4605c7c97e1c5579 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 23 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498248:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71800/0x4605c7c97e1c5580 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 24 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498254:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71800/0x4605c7c97e1c5580 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498281:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70200/0x4605c7c97e1c5587 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 25 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498288:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70200/0x4605c7c97e1c5587 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498332:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71400/0x4605c7c97e1c558e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 26 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498338:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71400/0x4605c7c97e1c558e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 26 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498356:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70400/0x4605c7c97e1c5595 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 27 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498362:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70400/0x4605c7c97e1c5595 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 27 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498377:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71200/0x4605c7c97e1c559c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 28 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498383:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71200/0x4605c7c97e1c559c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 28 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498398:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71c00/0x4605c7c97e1c55a3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 29 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498405:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71c00/0x4605c7c97e1c55a3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 29 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498421:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70a00/0x4605c7c97e1c55aa lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 30 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498428:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70a00/0x4605c7c97e1c55aa lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 30 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498442:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71600/0x4605c7c97e1c55b1 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 31 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498449:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71600/0x4605c7c97e1c55b1 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 31 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498466:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70600/0x4605c7c97e1c55b8 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 32 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498472:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70600/0x4605c7c97e1c55b8 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 32 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498488:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a000/0x4605c7c97e1c55bf lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 33 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498494:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a000/0x4605c7c97e1c55bf lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 33 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498512:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0bc00/0x4605c7c97e1c55c6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 34 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498518:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0bc00/0x4605c7c97e1c55c6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 34 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498533:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a200/0x4605c7c97e1c55cd lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 35 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498539:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a200/0x4605c7c97e1c55cd lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 35 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498566:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b200/0x4605c7c97e1c55d4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 36 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498573:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b200/0x4605c7c97e1c55d4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 36 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498588:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0be00/0x4605c7c97e1c55db lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 37 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498594:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0be00/0x4605c7c97e1c55db lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 37 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498609:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a800/0x4605c7c97e1c55e2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 38 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498616:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a800/0x4605c7c97e1c55e2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 38 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498630:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ae00/0x4605c7c97e1c55e9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 39 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498636:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ae00/0x4605c7c97e1c55e9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 39 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498667:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a600/0x4605c7c97e1c55f0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 40 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498673:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a600/0x4605c7c97e1c55f0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 40 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498689:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b800/0x4605c7c97e1c55f7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 41 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498695:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b800/0x4605c7c97e1c55f7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 41 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498724:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ba00/0x4605c7c97e1c55fe lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 42 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498732:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ba00/0x4605c7c97e1c55fe lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 42 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.498748:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0aa00/0x4605c7c97e1c5605 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 43 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.499158:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.499171:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:1.0:1788201389.499186:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788201389.499197:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x41c:0x0] updating i_size 0 i_blocks 1 00000020:00010000:1.0:1788201389.499202:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235844f530 (ffff9023500e7a28/0) at cl_lock_release():245 00000020:00010000:1.0:1788201389.499205:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235844f530 (ffff9023500e7a28/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201389.499209:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902345045c30 (ffff9023500e7e18/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788201389.499213:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235844f530 (ffff9023500e7a28/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201389.499216:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902345045c30 (ffff9023500e7a28/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201389.499245:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41c:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201389.499294:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.499305:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.499314:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00010000:00010000:1.0:1788201389.499325:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00000080:00200000:1.0:1788201389.499332:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.499337:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.499340:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.499344:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:1.0:1788201389.499348:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00010000:00010000:1.0:1788201389.499355:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00000080:00200000:1.0:1788201389.499365:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.499370:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:1.0:1788201389.499374:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.499380:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-17, flags=0 00000080:00200000:1.0:1788201389.499383:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201389.499387:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-17, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:0.0:1788201389.499471:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0aa00/0x4605c7c97e1c5605 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 44 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.499518:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201389.499543:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x41c:0x0]: losing layout lock 00010000:00010000:1.0:1788201389.499649:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201389.499657:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201389.499660:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c53ea lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x41c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d55a3 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:1.0:1788201389.499682:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201389.499695:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902345832300 x1875064442745856/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201389.499720:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345832300 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442745856:192.168.201.130@tcp:36:rm.0 00000100:00100000:1.0:1788201389.499972:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358723f00 going to sleep for 11 seconds 00010000:00010000:0.0:1788201389.501292:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b600/0x4605c7c97e1c560c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 44 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.501307:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b600/0x4605c7c97e1c560c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 44 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.501443:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5613 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 45 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.501450:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5613 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 45 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.501463:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c561a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 46 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.501469:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c561a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 46 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.501481:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c5621 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 47 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.501487:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c5621 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 47 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.501500:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5628 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 48 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.501506:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5628 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 48 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.502003:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c562f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 49 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.502012:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c562f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 49 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.502037:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c5636 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 50 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.502044:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c5636 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 50 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.502134:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c563d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 51 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.502142:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c563d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 51 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.502291:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c5644 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 52 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.502298:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c5644 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 52 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.502317:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c564b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 53 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.502322:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c564b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 53 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.502336:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c5652 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 54 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.502342:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c5652 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 54 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201389.502445:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902345831180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442745984:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:0.0:1788201389.504419:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6600/0x4605c7c97e1c54c3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 54 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.504434:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6600/0x4605c7c97e1c54c3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 54 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.504441:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6600/0x4605c7c97e1c54c3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 54 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.504448:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6600/0x4605c7c97e1c54c3 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 54 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.504456:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6600/0x4605c7c97e1c54c3 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 54 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.504462:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342ff6600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.504466:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6600/0x4605c7c97e1c54c3 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 54 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.504473:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6600/0x4605c7c97e1c54c3 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 54 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.504480:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6600/0x4605c7c97e1c54c3 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 54 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.504488:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.504493:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6600/0x4605c7c97e1c54c3 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 54 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.504500:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6600/0x4605c7c97e1c54c3 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 54 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.504696:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7e00/0x4605c7c97e1c54ca lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 53 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.504705:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7e00/0x4605c7c97e1c54ca lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 53 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.504710:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7e00/0x4605c7c97e1c54ca lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 53 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505664:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7e00/0x4605c7c97e1c54ca lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 53 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505678:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7e00/0x4605c7c97e1c54ca lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 53 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505684:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342ff7e00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.505689:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7e00/0x4605c7c97e1c54ca lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 53 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505695:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7e00/0x4605c7c97e1c54ca lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 53 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505702:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7e00/0x4605c7c97e1c54ca lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 53 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505709:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.505714:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7e00/0x4605c7c97e1c54ca lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 53 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505721:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7e00/0x4605c7c97e1c54ca lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 53 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505740:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6e00/0x4605c7c97e1c54d1 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 52 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505746:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6e00/0x4605c7c97e1c54d1 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 52 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505828:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6e00/0x4605c7c97e1c54d1 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 52 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505836:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6e00/0x4605c7c97e1c54d1 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 52 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505842:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6e00/0x4605c7c97e1c54d1 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 52 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505847:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342ff6e00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.505851:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6e00/0x4605c7c97e1c54d1 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 52 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505857:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6e00/0x4605c7c97e1c54d1 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 52 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505863:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6e00/0x4605c7c97e1c54d1 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 52 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505868:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.505871:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6e00/0x4605c7c97e1c54d1 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 52 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505946:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6e00/0x4605c7c97e1c54d1 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 52 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505962:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6200/0x4605c7c97e1c54d8 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 51 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505968:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6200/0x4605c7c97e1c54d8 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 51 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505974:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6200/0x4605c7c97e1c54d8 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 51 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505979:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6200/0x4605c7c97e1c54d8 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 51 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505984:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6200/0x4605c7c97e1c54d8 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 51 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505988:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342ff6200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.505992:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6200/0x4605c7c97e1c54d8 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 51 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.505997:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6200/0x4605c7c97e1c54d8 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 51 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506003:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6200/0x4605c7c97e1c54d8 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 51 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506009:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.506012:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6200/0x4605c7c97e1c54d8 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 51 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506018:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6200/0x4605c7c97e1c54d8 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 51 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506030:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6c00/0x4605c7c97e1c54df lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 50 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506036:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6c00/0x4605c7c97e1c54df lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 50 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506041:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6c00/0x4605c7c97e1c54df lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 50 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506047:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6c00/0x4605c7c97e1c54df lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 50 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506052:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6c00/0x4605c7c97e1c54df lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 50 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506057:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342ff6c00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.506130:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6c00/0x4605c7c97e1c54df lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 50 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506137:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6c00/0x4605c7c97e1c54df lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 50 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506143:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6c00/0x4605c7c97e1c54df lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 50 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506148:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.506152:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6c00/0x4605c7c97e1c54df lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 50 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506342:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6c00/0x4605c7c97e1c54df lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 50 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506357:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7000/0x4605c7c97e1c54e6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 49 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506363:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7000/0x4605c7c97e1c54e6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 49 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506368:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7000/0x4605c7c97e1c54e6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 49 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506374:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7000/0x4605c7c97e1c54e6 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 49 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506380:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7000/0x4605c7c97e1c54e6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 49 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506385:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342ff7000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.506388:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7000/0x4605c7c97e1c54e6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 49 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506394:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7000/0x4605c7c97e1c54e6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 49 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506400:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7000/0x4605c7c97e1c54e6 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 49 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506406:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.506410:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7000/0x4605c7c97e1c54e6 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 49 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506416:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7000/0x4605c7c97e1c54e6 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 49 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506429:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7a00/0x4605c7c97e1c54ed lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 48 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506436:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7a00/0x4605c7c97e1c54ed lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 48 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506443:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7a00/0x4605c7c97e1c54ed lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 48 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506449:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7a00/0x4605c7c97e1c54ed lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 48 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506456:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7a00/0x4605c7c97e1c54ed lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 48 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506462:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342ff7a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.506465:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7a00/0x4605c7c97e1c54ed lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 48 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506471:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7a00/0x4605c7c97e1c54ed lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 48 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506476:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7a00/0x4605c7c97e1c54ed lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 48 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506481:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.506484:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7a00/0x4605c7c97e1c54ed lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 48 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506489:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7a00/0x4605c7c97e1c54ed lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 48 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506500:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6400/0x4605c7c97e1c54f4 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 47 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506507:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6400/0x4605c7c97e1c54f4 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 47 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506512:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6400/0x4605c7c97e1c54f4 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 47 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506518:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6400/0x4605c7c97e1c54f4 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 47 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506536:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6400/0x4605c7c97e1c54f4 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 47 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506541:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342ff6400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.506543:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6400/0x4605c7c97e1c54f4 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 47 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506704:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6400/0x4605c7c97e1c54f4 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 47 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506713:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6400/0x4605c7c97e1c54f4 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 47 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506718:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.506722:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6400/0x4605c7c97e1c54f4 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 47 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506727:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff6400/0x4605c7c97e1c54f4 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 47 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506742:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7600/0x4605c7c97e1c54fb lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 46 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506747:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7600/0x4605c7c97e1c54fb lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 46 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506827:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7600/0x4605c7c97e1c54fb lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 46 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506833:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7600/0x4605c7c97e1c54fb lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 46 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506839:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7600/0x4605c7c97e1c54fb lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 46 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506843:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342ff7600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.506846:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7600/0x4605c7c97e1c54fb lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 46 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506851:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7600/0x4605c7c97e1c54fb lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 46 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506856:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7600/0x4605c7c97e1c54fb lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 46 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506861:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.506863:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7600/0x4605c7c97e1c54fb lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 46 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506868:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7600/0x4605c7c97e1c54fb lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 46 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506915:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7400/0x4605c7c97e1c5502 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 45 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506921:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7400/0x4605c7c97e1c5502 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 45 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506972:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7400/0x4605c7c97e1c5502 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 45 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506980:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7400/0x4605c7c97e1c5502 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 45 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506986:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7400/0x4605c7c97e1c5502 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 45 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506990:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342ff7400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.506993:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7400/0x4605c7c97e1c5502 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 45 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.506999:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7400/0x4605c7c97e1c5502 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 45 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507005:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7400/0x4605c7c97e1c5502 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 45 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507010:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.507014:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7400/0x4605c7c97e1c5502 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 45 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507019:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7400/0x4605c7c97e1c5502 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 45 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507032:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7200/0x4605c7c97e1c5509 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 44 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507038:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7200/0x4605c7c97e1c5509 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 44 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507044:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7200/0x4605c7c97e1c5509 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 44 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507050:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7200/0x4605c7c97e1c5509 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 44 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507055:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7200/0x4605c7c97e1c5509 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 44 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507059:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342ff7200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.507133:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7200/0x4605c7c97e1c5509 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 44 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507140:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7200/0x4605c7c97e1c5509 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 44 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507146:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7200/0x4605c7c97e1c5509 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 44 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507152:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.507354:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7200/0x4605c7c97e1c5509 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 44 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507363:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342ff7200/0x4605c7c97e1c5509 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 44 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507377:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b800/0x4605c7c97e1c5510 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 43 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507384:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b800/0x4605c7c97e1c5510 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 43 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507389:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b800/0x4605c7c97e1c5510 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 43 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507395:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b800/0x4605c7c97e1c5510 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 43 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507401:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b800/0x4605c7c97e1c5510 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 43 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507407:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865b800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.507410:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b800/0x4605c7c97e1c5510 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 43 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507415:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b800/0x4605c7c97e1c5510 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 43 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507421:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b800/0x4605c7c97e1c5510 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 43 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507426:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.507430:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b800/0x4605c7c97e1c5510 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 43 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507436:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b800/0x4605c7c97e1c5510 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 43 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507448:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b600/0x4605c7c97e1c5517 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 42 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507454:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b600/0x4605c7c97e1c5517 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 42 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507458:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b600/0x4605c7c97e1c5517 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 42 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507463:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b600/0x4605c7c97e1c5517 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 42 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507468:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b600/0x4605c7c97e1c5517 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 42 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507472:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865b600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.507474:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b600/0x4605c7c97e1c5517 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 42 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507479:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b600/0x4605c7c97e1c5517 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 42 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507484:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b600/0x4605c7c97e1c5517 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 42 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507489:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.507492:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b600/0x4605c7c97e1c5517 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 42 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507498:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b600/0x4605c7c97e1c5517 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 42 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507509:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c551e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 41 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507515:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c551e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 41 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507521:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c551e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 41 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507655:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c551e lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 41 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507662:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c551e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 41 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507668:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865a200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.507672:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c551e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 41 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507677:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c551e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 41 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507683:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c551e lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 41 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507688:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.507691:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c551e lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 41 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507696:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c551e lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 41 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507712:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a000/0x4605c7c97e1c5525 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 40 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507717:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a000/0x4605c7c97e1c5525 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 40 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507722:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a000/0x4605c7c97e1c5525 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 40 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507726:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a000/0x4605c7c97e1c5525 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 40 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507731:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a000/0x4605c7c97e1c5525 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 40 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507736:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865a000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.507738:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a000/0x4605c7c97e1c5525 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 40 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507743:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a000/0x4605c7c97e1c5525 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 40 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507748:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a000/0x4605c7c97e1c5525 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 40 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507826:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.507831:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a000/0x4605c7c97e1c5525 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 40 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507837:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a000/0x4605c7c97e1c5525 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 40 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507850:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c552c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 39 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507856:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c552c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 39 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507861:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c552c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 39 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507867:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c552c lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 39 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.507873:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c552c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 39 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508005:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865a400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.508008:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c552c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 39 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508014:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c552c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 39 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508020:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c552c lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 39 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508026:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.508029:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c552c lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 39 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508035:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c552c lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 39 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508050:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c5533 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 38 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508056:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c5533 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 38 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508134:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c5533 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 38 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508140:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c5533 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 38 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508146:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c5533 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 38 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508152:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865aa00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.508155:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c5533 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 38 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508286:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c5533 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 38 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508294:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c5533 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 38 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508299:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.508303:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c5533 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 38 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508309:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c5533 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 38 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508323:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c553a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 37 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508329:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c553a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 37 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508335:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c553a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 37 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508341:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c553a lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 37 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508347:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c553a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 37 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508352:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865b400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.508355:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c553a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 37 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508359:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c553a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 37 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508365:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c553a lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 37 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508369:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.508372:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c553a lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 37 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508378:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c553a lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 37 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508390:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5541 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 36 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508397:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5541 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 36 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508403:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5541 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 36 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508409:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5541 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 36 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508414:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5541 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 36 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508419:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865ac00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.508422:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5541 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 36 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508427:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5541 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 36 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508433:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5541 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 36 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508439:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.508442:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5541 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 36 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.508448:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5541 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 36 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201389.508474:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442744832:192.168.201.130@tcp:63:ll_sa.0 00000100:00100000:1.0:1788201389.518209:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345832300 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442745856:192.168.201.130@tcp:36:rm.0 00000080:00200000:1.0:1788201389.518305:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201389.518319:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201389.518378:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201389.518395:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x41c:0x0](ffff9023449b3248) 00000008:00010000:1.0:1788201389.518434:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201389.518526:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.518545:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.518973:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00010000:00010000:1.0:1788201389.518988:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00000080:00200000:1.0:1788201389.518996:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.519003:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.519006:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.519014:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:1.0:1788201389.519018:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00010000:00010000:1.0:1788201389.519031:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00000080:00200000:1.0:1788201389.519045:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.519052:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:1.0:1788201389.519067:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.519071:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-18, flags=8 00010000:00010000:1.0:1788201389.519080:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1788201389.519087:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53f1 from it ffffa0b181c3baf8 00010000:00010000:1.0:1788201389.519090:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201389.519096:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:0.0:1788201389.519470:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-18:[0x200000bd1:0x41d:0x0](ffff9023449a00c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201389.519477:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x41d:0x0](ffff9023449a00c8),name=t-18 00000002:00010000:0.0:1788201389.519490:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x41d:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.519498:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.519507:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201389.519513:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.519518:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x41d:0x0](ffff9023449a00c8) 00000080:00010000:0.0:1788201389.519521:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x41d:0x0](ffff9023449a00c8) for lock 0x4605c7c97e1c53f1 00000080:00010000:0.0:1788201389.519525:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53f1 from it ffffa0b181c3bcb8 00010000:00010000:0.0:1788201389.519529:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.519536:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.519583:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.519591:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x41d:0x0] 00000008:00010000:0.0:1788201389.519615:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201389.519619:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023500506b0, osc lock ffff902343fb7db0, flags 0x400001002 00010000:00010000:0.0:1788201389.519637:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.519653:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1788201389.519667:0:17859:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1788201389.519678:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788201389.519682:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.519689:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1788201389.519700:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788201389.519710:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x41d:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201389.519714:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.519717:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.519720:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023500506b0 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.519724:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.519726:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023500506b0 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.519751:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41d:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.519784:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.519792:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.519800:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00010000:00010000:0.0:1788201389.519809:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00000080:00200000:0.0:1788201389.519815:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.519820:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.519822:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.519826:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:0.0:1788201389.519829:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00010000:00010000:0.0:1788201389.519835:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00000080:00200000:0.0:1788201389.519843:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.519847:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.519850:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.519855:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-18, flags=0 00000080:00200000:0.0:1788201389.519859:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.519863:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-18, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:0.0:1788201389.519925:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.519947:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x41d:0x0]: losing layout lock 00010000:00010000:0.0:1788201389.519991:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201389.519998:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201389.520002:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730600/0x4605c7c97e1c53f1 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x41d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d55c6 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:0.0:1788201389.520024:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201389.520036:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450b9500 x1875064442746240/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201389.520062:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442746240:192.168.201.130@tcp:36:rm.0 00010000:00010000:1.0:1788201389.520208:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5659 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 36 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520217:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5659 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 36 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520241:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c5660 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 37 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520247:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c5660 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 37 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520280:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5667 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 38 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520285:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5667 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 38 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520302:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c566e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 39 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520308:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c566e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 39 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520322:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124c00/0x4605c7c97e1c5675 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 40 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520327:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124c00/0x4605c7c97e1c5675 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 40 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520343:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125c00/0x4605c7c97e1c567c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 41 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520349:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125c00/0x4605c7c97e1c567c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 41 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520366:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125000/0x4605c7c97e1c5683 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 42 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520372:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125000/0x4605c7c97e1c5683 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 42 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520387:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125600/0x4605c7c97e1c568a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 43 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520394:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125600/0x4605c7c97e1c568a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 43 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520409:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124600/0x4605c7c97e1c5691 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 44 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520415:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124600/0x4605c7c97e1c5691 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 44 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520430:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125e00/0x4605c7c97e1c5698 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 45 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520438:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125e00/0x4605c7c97e1c5698 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 45 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520453:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125800/0x4605c7c97e1c569f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 46 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520458:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125800/0x4605c7c97e1c569f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 46 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520473:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124400/0x4605c7c97e1c56a6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 47 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520479:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124400/0x4605c7c97e1c56a6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 47 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520494:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125400/0x4605c7c97e1c56ad lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 48 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520501:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125400/0x4605c7c97e1c56ad lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 48 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520516:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125200/0x4605c7c97e1c56b4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 49 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520522:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125200/0x4605c7c97e1c56b4 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 49 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520539:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125a00/0x4605c7c97e1c56bb lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 50 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520545:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125a00/0x4605c7c97e1c56bb lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 50 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520562:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64e00/0x4605c7c97e1c56c2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 51 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520568:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64e00/0x4605c7c97e1c56c2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 51 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520604:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65200/0x4605c7c97e1c56c9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 52 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520611:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65200/0x4605c7c97e1c56c9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 52 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520629:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65c00/0x4605c7c97e1c56d0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 53 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520635:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65c00/0x4605c7c97e1c56d0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 53 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520650:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64800/0x4605c7c97e1c56d7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 54 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520656:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64800/0x4605c7c97e1c56d7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 54 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520673:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65800/0x4605c7c97e1c56de lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 55 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520679:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65800/0x4605c7c97e1c56de lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 55 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520694:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65a00/0x4605c7c97e1c56e5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 56 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520701:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65a00/0x4605c7c97e1c56e5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 56 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520716:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65600/0x4605c7c97e1c56ec lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 57 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520722:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65600/0x4605c7c97e1c56ec lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 57 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520737:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65e00/0x4605c7c97e1c56f3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 58 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520743:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65e00/0x4605c7c97e1c56f3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 58 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520758:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64000/0x4605c7c97e1c56fa lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 59 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520764:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64000/0x4605c7c97e1c56fa lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 59 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520780:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64200/0x4605c7c97e1c5701 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 60 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520785:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64200/0x4605c7c97e1c5701 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 60 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520797:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64a00/0x4605c7c97e1c5708 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 61 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520802:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64a00/0x4605c7c97e1c5708 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 61 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520817:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c000/0x4605c7c97e1c570f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 62 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520822:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c000/0x4605c7c97e1c570f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 62 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520836:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d200/0x4605c7c97e1c5716 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 63 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520843:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d200/0x4605c7c97e1c5716 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 63 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520860:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869dc00/0x4605c7c97e1c571d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 64 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520866:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869dc00/0x4605c7c97e1c571d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 64 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520905:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869da00/0x4605c7c97e1c5724 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 65 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520912:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869da00/0x4605c7c97e1c5724 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 65 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520930:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c200/0x4605c7c97e1c572b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 66 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520936:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c200/0x4605c7c97e1c572b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 66 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520950:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869de00/0x4605c7c97e1c5732 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 67 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520956:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869de00/0x4605c7c97e1c5732 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 67 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520987:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d600/0x4605c7c97e1c5739 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 68 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.520993:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d600/0x4605c7c97e1c5739 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 68 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521009:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c800/0x4605c7c97e1c5740 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 69 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521015:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c800/0x4605c7c97e1c5740 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 69 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521030:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869cc00/0x4605c7c97e1c5747 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 70 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521036:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869cc00/0x4605c7c97e1c5747 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 70 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521051:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d800/0x4605c7c97e1c574e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 71 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521056:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d800/0x4605c7c97e1c574e lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 71 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521072:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c600/0x4605c7c97e1c5755 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 72 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521077:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c600/0x4605c7c97e1c5755 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 72 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521092:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869ce00/0x4605c7c97e1c575c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 73 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521098:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869ce00/0x4605c7c97e1c575c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 73 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521115:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733000/0x4605c7c97e1c5763 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 74 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521121:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733000/0x4605c7c97e1c5763 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 74 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521136:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732a00/0x4605c7c97e1c576a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 75 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521142:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732a00/0x4605c7c97e1c576a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 75 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521159:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733a00/0x4605c7c97e1c5771 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 76 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521165:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733a00/0x4605c7c97e1c5771 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 76 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521182:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732600/0x4605c7c97e1c5778 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 77 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521188:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732600/0x4605c7c97e1c5778 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 77 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521202:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732400/0x4605c7c97e1c577f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 78 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521208:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732400/0x4605c7c97e1c577f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 78 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521223:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733600/0x4605c7c97e1c5786 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 79 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521228:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733600/0x4605c7c97e1c5786 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 79 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521242:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733e00/0x4605c7c97e1c578d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 80 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521247:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733e00/0x4605c7c97e1c578d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 80 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521259:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733800/0x4605c7c97e1c5794 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 81 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521265:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733800/0x4605c7c97e1c5794 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 81 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521294:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732800/0x4605c7c97e1c579b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 82 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521300:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732800/0x4605c7c97e1c579b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 82 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521313:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732e00/0x4605c7c97e1c57a2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 83 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521318:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732e00/0x4605c7c97e1c57a2 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 83 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521346:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733400/0x4605c7c97e1c57a9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 84 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521353:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733400/0x4605c7c97e1c57a9 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 84 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521369:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732000/0x4605c7c97e1c57b0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 85 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521375:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732000/0x4605c7c97e1c57b0 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 85 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521393:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756c00/0x4605c7c97e1c57b7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 86 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521400:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756c00/0x4605c7c97e1c57b7 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 86 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521418:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757c00/0x4605c7c97e1c57be lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 87 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521423:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757c00/0x4605c7c97e1c57be lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 87 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521438:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757200/0x4605c7c97e1c57c5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 88 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521445:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757200/0x4605c7c97e1c57c5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 88 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521459:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757800/0x4605c7c97e1c57cc lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 89 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521465:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757800/0x4605c7c97e1c57cc lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 89 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521479:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757a00/0x4605c7c97e1c57d3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 90 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521485:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757a00/0x4605c7c97e1c57d3 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 90 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521500:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756e00/0x4605c7c97e1c57da lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 91 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521507:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756e00/0x4605c7c97e1c57da lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 91 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521522:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756800/0x4605c7c97e1c57e1 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 92 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521529:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756800/0x4605c7c97e1c57e1 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 92 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521545:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757e00/0x4605c7c97e1c57e8 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 93 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521552:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757e00/0x4605c7c97e1c57e8 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 93 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521568:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756400/0x4605c7c97e1c57ef lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 94 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521574:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756400/0x4605c7c97e1c57ef lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 94 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521594:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757400/0x4605c7c97e1c57f6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 95 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521600:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757400/0x4605c7c97e1c57f6 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 95 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521618:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756600/0x4605c7c97e1c57fd lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 96 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521624:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756600/0x4605c7c97e1c57fd lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 96 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521639:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757000/0x4605c7c97e1c5804 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 97 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521645:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757000/0x4605c7c97e1c5804 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 97 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521663:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d800/0x4605c7c97e1c580b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 98 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521669:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d800/0x4605c7c97e1c580b lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 98 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521684:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5812 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 99 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.521690:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5812 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 99 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.521781:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442746368:192.168.201.130@tcp:63:ll_sa.0 00000100:00100000:0.0:1788201389.531056:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436c09c0 going to sleep for 11 seconds 00010000:00010000:0.0:1788201389.531472:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5819 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 100 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.531484:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5819 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 100 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201389.531793:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442746496:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:1.0:1788201389.546859:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70c00/0x4605c7c97e1c5572 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 100 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.546914:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70c00/0x4605c7c97e1c5572 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 100 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.546921:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70c00/0x4605c7c97e1c5572 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 100 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.546927:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70c00/0x4605c7c97e1c5572 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 100 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.546935:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70c00/0x4605c7c97e1c5572 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 100 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.546941:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342e70c00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.546945:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70c00/0x4605c7c97e1c5572 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 100 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.546952:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70c00/0x4605c7c97e1c5572 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 100 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.546960:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70c00/0x4605c7c97e1c5572 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 100 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.546967:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.546973:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70c00/0x4605c7c97e1c5572 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 100 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.546978:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70c00/0x4605c7c97e1c5572 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 100 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.546999:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71a00/0x4605c7c97e1c5579 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 99 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547006:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71a00/0x4605c7c97e1c5579 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 99 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547011:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71a00/0x4605c7c97e1c5579 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 99 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547017:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71a00/0x4605c7c97e1c5579 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 99 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547023:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71a00/0x4605c7c97e1c5579 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 99 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547028:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342e71a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.547031:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71a00/0x4605c7c97e1c5579 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 99 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547036:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71a00/0x4605c7c97e1c5579 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 99 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547042:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71a00/0x4605c7c97e1c5579 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 99 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547047:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.547051:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71a00/0x4605c7c97e1c5579 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 99 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547056:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71a00/0x4605c7c97e1c5579 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 99 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547080:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71800/0x4605c7c97e1c5580 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 98 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547086:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71800/0x4605c7c97e1c5580 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 98 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547091:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71800/0x4605c7c97e1c5580 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 98 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547097:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71800/0x4605c7c97e1c5580 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 98 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547104:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71800/0x4605c7c97e1c5580 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 98 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547108:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342e71800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.547111:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71800/0x4605c7c97e1c5580 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 98 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547116:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71800/0x4605c7c97e1c5580 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 98 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547122:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71800/0x4605c7c97e1c5580 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 98 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547127:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.547130:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71800/0x4605c7c97e1c5580 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 98 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547135:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71800/0x4605c7c97e1c5580 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 98 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547146:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70200/0x4605c7c97e1c5587 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 97 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547152:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70200/0x4605c7c97e1c5587 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 97 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547639:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70200/0x4605c7c97e1c5587 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 97 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547646:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70200/0x4605c7c97e1c5587 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 97 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547652:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70200/0x4605c7c97e1c5587 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 97 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547659:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342e70200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.547662:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70200/0x4605c7c97e1c5587 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 97 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547668:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70200/0x4605c7c97e1c5587 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 97 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547675:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70200/0x4605c7c97e1c5587 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 97 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547681:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.547685:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70200/0x4605c7c97e1c5587 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 97 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547690:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70200/0x4605c7c97e1c5587 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 97 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547705:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71400/0x4605c7c97e1c558e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 96 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547711:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71400/0x4605c7c97e1c558e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 96 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547716:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71400/0x4605c7c97e1c558e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 96 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547721:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71400/0x4605c7c97e1c558e lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 96 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547727:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71400/0x4605c7c97e1c558e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 96 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547732:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342e71400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.547735:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71400/0x4605c7c97e1c558e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 96 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547741:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71400/0x4605c7c97e1c558e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 96 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547747:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71400/0x4605c7c97e1c558e lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 96 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547753:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.547756:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71400/0x4605c7c97e1c558e lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 96 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547762:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71400/0x4605c7c97e1c558e lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 96 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547773:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70400/0x4605c7c97e1c5595 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 95 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547779:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70400/0x4605c7c97e1c5595 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 95 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547784:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70400/0x4605c7c97e1c5595 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 95 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547789:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70400/0x4605c7c97e1c5595 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 95 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547796:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70400/0x4605c7c97e1c5595 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 95 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547802:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342e70400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.547804:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70400/0x4605c7c97e1c5595 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 95 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547810:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70400/0x4605c7c97e1c5595 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 95 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547816:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70400/0x4605c7c97e1c5595 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 95 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547821:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.547825:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70400/0x4605c7c97e1c5595 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 95 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547831:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70400/0x4605c7c97e1c5595 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 95 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547841:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71200/0x4605c7c97e1c559c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 94 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547847:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71200/0x4605c7c97e1c559c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 94 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547853:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71200/0x4605c7c97e1c559c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 94 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547859:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71200/0x4605c7c97e1c559c lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 94 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547864:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71200/0x4605c7c97e1c559c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 94 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547869:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342e71200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.547872:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71200/0x4605c7c97e1c559c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 94 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547949:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71200/0x4605c7c97e1c559c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 94 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547956:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71200/0x4605c7c97e1c559c lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 94 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547961:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.547965:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71200/0x4605c7c97e1c559c lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 94 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547971:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71200/0x4605c7c97e1c559c lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 94 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547984:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71c00/0x4605c7c97e1c55a3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 93 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547990:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71c00/0x4605c7c97e1c55a3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 93 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.547996:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71c00/0x4605c7c97e1c55a3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 93 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548002:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71c00/0x4605c7c97e1c55a3 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 93 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548008:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71c00/0x4605c7c97e1c55a3 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 93 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548014:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342e71c00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.548017:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71c00/0x4605c7c97e1c55a3 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 93 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548023:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71c00/0x4605c7c97e1c55a3 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 93 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548029:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71c00/0x4605c7c97e1c55a3 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 93 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548034:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.548037:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71c00/0x4605c7c97e1c55a3 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 93 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548043:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71c00/0x4605c7c97e1c55a3 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 93 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548054:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70a00/0x4605c7c97e1c55aa lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 92 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548124:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70a00/0x4605c7c97e1c55aa lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 92 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548131:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70a00/0x4605c7c97e1c55aa lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 92 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548137:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70a00/0x4605c7c97e1c55aa lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 92 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548143:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70a00/0x4605c7c97e1c55aa lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 92 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548148:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342e70a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.548151:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70a00/0x4605c7c97e1c55aa lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 92 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548276:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70a00/0x4605c7c97e1c55aa lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 92 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548284:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70a00/0x4605c7c97e1c55aa lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 92 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548289:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.548293:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70a00/0x4605c7c97e1c55aa lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 92 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548300:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70a00/0x4605c7c97e1c55aa lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 92 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548314:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71600/0x4605c7c97e1c55b1 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 91 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548320:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71600/0x4605c7c97e1c55b1 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 91 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548326:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71600/0x4605c7c97e1c55b1 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 91 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548332:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71600/0x4605c7c97e1c55b1 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 91 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548337:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71600/0x4605c7c97e1c55b1 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 91 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548507:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342e71600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.548511:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71600/0x4605c7c97e1c55b1 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 91 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548518:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71600/0x4605c7c97e1c55b1 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 91 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548639:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71600/0x4605c7c97e1c55b1 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 91 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548647:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.548651:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71600/0x4605c7c97e1c55b1 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 91 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548658:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e71600/0x4605c7c97e1c55b1 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 91 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548673:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70600/0x4605c7c97e1c55b8 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 90 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548679:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70600/0x4605c7c97e1c55b8 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 90 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548684:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70600/0x4605c7c97e1c55b8 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 90 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548690:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70600/0x4605c7c97e1c55b8 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 90 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548695:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70600/0x4605c7c97e1c55b8 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 90 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548700:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342e70600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.548703:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70600/0x4605c7c97e1c55b8 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 90 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548709:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70600/0x4605c7c97e1c55b8 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 90 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548715:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70600/0x4605c7c97e1c55b8 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 90 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548721:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.548724:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70600/0x4605c7c97e1c55b8 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 90 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548730:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342e70600/0x4605c7c97e1c55b8 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 90 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548740:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a000/0x4605c7c97e1c55bf lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 89 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548745:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a000/0x4605c7c97e1c55bf lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 89 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548750:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a000/0x4605c7c97e1c55bf lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 89 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548755:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a000/0x4605c7c97e1c55bf lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 89 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548760:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a000/0x4605c7c97e1c55bf lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 89 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548765:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342f0a000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.548767:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a000/0x4605c7c97e1c55bf lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 89 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548773:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a000/0x4605c7c97e1c55bf lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 89 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548779:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a000/0x4605c7c97e1c55bf lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 89 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548785:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.548788:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a000/0x4605c7c97e1c55bf lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 89 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548794:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a000/0x4605c7c97e1c55bf lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 89 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548805:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0bc00/0x4605c7c97e1c55c6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 88 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548812:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0bc00/0x4605c7c97e1c55c6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 88 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548818:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0bc00/0x4605c7c97e1c55c6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 88 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548823:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0bc00/0x4605c7c97e1c55c6 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 88 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548829:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0bc00/0x4605c7c97e1c55c6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 88 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548834:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342f0bc00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.548836:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0bc00/0x4605c7c97e1c55c6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 88 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548841:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0bc00/0x4605c7c97e1c55c6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 88 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548846:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0bc00/0x4605c7c97e1c55c6 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 88 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548851:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.548854:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0bc00/0x4605c7c97e1c55c6 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 88 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548859:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0bc00/0x4605c7c97e1c55c6 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 88 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548870:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a200/0x4605c7c97e1c55cd lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 87 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548963:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a200/0x4605c7c97e1c55cd lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 87 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548970:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a200/0x4605c7c97e1c55cd lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 87 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548975:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a200/0x4605c7c97e1c55cd lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 87 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548981:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a200/0x4605c7c97e1c55cd lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 87 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548986:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342f0a200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.548989:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a200/0x4605c7c97e1c55cd lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 87 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.548995:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a200/0x4605c7c97e1c55cd lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 87 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549002:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a200/0x4605c7c97e1c55cd lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 87 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549007:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.549011:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a200/0x4605c7c97e1c55cd lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 87 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549017:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a200/0x4605c7c97e1c55cd lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 87 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549030:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b200/0x4605c7c97e1c55d4 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 86 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549035:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b200/0x4605c7c97e1c55d4 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 86 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549040:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b200/0x4605c7c97e1c55d4 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 86 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549046:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b200/0x4605c7c97e1c55d4 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 86 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549051:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b200/0x4605c7c97e1c55d4 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 86 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549057:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342f0b200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.549128:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b200/0x4605c7c97e1c55d4 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 86 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549137:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b200/0x4605c7c97e1c55d4 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 86 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549143:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b200/0x4605c7c97e1c55d4 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 86 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549147:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.549151:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b200/0x4605c7c97e1c55d4 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 86 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549276:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b200/0x4605c7c97e1c55d4 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 86 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549291:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0be00/0x4605c7c97e1c55db lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 85 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549297:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0be00/0x4605c7c97e1c55db lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 85 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549303:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0be00/0x4605c7c97e1c55db lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 85 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549309:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0be00/0x4605c7c97e1c55db lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 85 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549315:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0be00/0x4605c7c97e1c55db lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 85 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549321:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342f0be00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.549324:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0be00/0x4605c7c97e1c55db lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 85 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549330:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0be00/0x4605c7c97e1c55db lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 85 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549336:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0be00/0x4605c7c97e1c55db lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 85 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549342:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.549345:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0be00/0x4605c7c97e1c55db lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 85 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549423:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0be00/0x4605c7c97e1c55db lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 85 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549437:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a800/0x4605c7c97e1c55e2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 84 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549443:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a800/0x4605c7c97e1c55e2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 84 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549446:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a800/0x4605c7c97e1c55e2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 84 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549449:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a800/0x4605c7c97e1c55e2 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 84 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549453:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a800/0x4605c7c97e1c55e2 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 84 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549455:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342f0a800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.549457:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a800/0x4605c7c97e1c55e2 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 84 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549461:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a800/0x4605c7c97e1c55e2 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 84 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549465:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a800/0x4605c7c97e1c55e2 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 84 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549468:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.549470:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a800/0x4605c7c97e1c55e2 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 84 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549473:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a800/0x4605c7c97e1c55e2 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 84 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549481:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ae00/0x4605c7c97e1c55e9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 83 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549485:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ae00/0x4605c7c97e1c55e9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 83 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549488:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ae00/0x4605c7c97e1c55e9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 83 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549491:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ae00/0x4605c7c97e1c55e9 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 83 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549494:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ae00/0x4605c7c97e1c55e9 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 83 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549497:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342f0ae00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.549498:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ae00/0x4605c7c97e1c55e9 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 83 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549501:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ae00/0x4605c7c97e1c55e9 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 83 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549505:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ae00/0x4605c7c97e1c55e9 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 83 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549508:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.549510:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ae00/0x4605c7c97e1c55e9 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 83 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549513:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ae00/0x4605c7c97e1c55e9 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 83 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549519:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a600/0x4605c7c97e1c55f0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 82 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549522:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a600/0x4605c7c97e1c55f0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 82 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549772:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a600/0x4605c7c97e1c55f0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 82 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549777:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a600/0x4605c7c97e1c55f0 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 82 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549782:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a600/0x4605c7c97e1c55f0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 82 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549786:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342f0a600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.549789:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a600/0x4605c7c97e1c55f0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 82 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549794:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a600/0x4605c7c97e1c55f0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 82 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549801:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a600/0x4605c7c97e1c55f0 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 82 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549807:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.549810:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a600/0x4605c7c97e1c55f0 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 82 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549816:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0a600/0x4605c7c97e1c55f0 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 82 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549829:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b800/0x4605c7c97e1c55f7 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 81 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549835:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b800/0x4605c7c97e1c55f7 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 81 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549841:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b800/0x4605c7c97e1c55f7 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 81 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549846:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b800/0x4605c7c97e1c55f7 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 81 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549852:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b800/0x4605c7c97e1c55f7 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 81 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549856:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342f0b800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.549858:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b800/0x4605c7c97e1c55f7 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 81 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549863:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b800/0x4605c7c97e1c55f7 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 81 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549869:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b800/0x4605c7c97e1c55f7 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 81 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549874:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.549902:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b800/0x4605c7c97e1c55f7 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 81 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549909:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b800/0x4605c7c97e1c55f7 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 81 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549920:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ba00/0x4605c7c97e1c55fe lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 80 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549925:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ba00/0x4605c7c97e1c55fe lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 80 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549931:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ba00/0x4605c7c97e1c55fe lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 80 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549936:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ba00/0x4605c7c97e1c55fe lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 80 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549942:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ba00/0x4605c7c97e1c55fe lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 80 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549947:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342f0ba00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.549950:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ba00/0x4605c7c97e1c55fe lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 80 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549954:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ba00/0x4605c7c97e1c55fe lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 80 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549958:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ba00/0x4605c7c97e1c55fe lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 80 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549962:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.549964:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ba00/0x4605c7c97e1c55fe lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 80 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549968:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0ba00/0x4605c7c97e1c55fe lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 80 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549976:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0aa00/0x4605c7c97e1c5605 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 79 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549980:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0aa00/0x4605c7c97e1c5605 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 79 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549983:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0aa00/0x4605c7c97e1c5605 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 79 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549986:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0aa00/0x4605c7c97e1c5605 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 79 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549990:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0aa00/0x4605c7c97e1c5605 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 79 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.549995:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342f0aa00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.549997:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0aa00/0x4605c7c97e1c5605 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 79 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550002:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0aa00/0x4605c7c97e1c5605 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 79 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550008:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0aa00/0x4605c7c97e1c5605 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 79 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550013:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.550015:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0aa00/0x4605c7c97e1c5605 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 79 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550020:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0aa00/0x4605c7c97e1c5605 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 79 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550031:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b600/0x4605c7c97e1c560c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 78 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550036:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b600/0x4605c7c97e1c560c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 78 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550042:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b600/0x4605c7c97e1c560c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 78 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550048:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b600/0x4605c7c97e1c560c lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 78 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550054:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b600/0x4605c7c97e1c560c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 78 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550058:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342f0b600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.550071:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b600/0x4605c7c97e1c560c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 78 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550077:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b600/0x4605c7c97e1c560c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 78 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550082:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b600/0x4605c7c97e1c560c lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 78 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550086:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.550090:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b600/0x4605c7c97e1c560c lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 78 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550096:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902342f0b600/0x4605c7c97e1c560c lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 78 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550109:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5613 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 77 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550114:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5613 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 77 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550119:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5613 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 77 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550124:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5613 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 77 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550129:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5613 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 77 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550134:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839de00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.550137:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5613 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 77 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550143:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5613 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 77 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550149:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5613 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 77 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550155:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.550307:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5613 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 77 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550314:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839de00/0x4605c7c97e1c5613 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 77 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550329:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c561a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 76 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550335:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c561a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 76 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550340:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c561a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 76 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550347:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c561a lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 76 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550352:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c561a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 76 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550357:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839dc00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.550360:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c561a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 76 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550365:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c561a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 76 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550371:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c561a lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 76 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550565:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.550569:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c561a lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 76 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550576:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839dc00/0x4605c7c97e1c561a lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 76 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550589:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c5621 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 75 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550595:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c5621 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 75 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550601:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c5621 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 75 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550615:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c5621 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 75 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550621:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c5621 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 75 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550627:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839d600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.550630:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c5621 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 75 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550636:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c5621 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 75 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550642:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c5621 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 75 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550646:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.550648:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c5621 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 75 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550652:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d600/0x4605c7c97e1c5621 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 75 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550661:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5628 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 74 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550665:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5628 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 74 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550668:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5628 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 74 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550671:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5628 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 74 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550674:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5628 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 74 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550677:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839cc00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.550679:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5628 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 74 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550682:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5628 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 74 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550685:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5628 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 74 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550689:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.550690:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5628 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 74 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550693:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839cc00/0x4605c7c97e1c5628 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 74 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550699:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c562f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 73 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550702:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c562f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 73 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550705:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c562f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 73 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550708:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c562f lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 73 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550711:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c562f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 73 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550714:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839c400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.550715:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c562f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 73 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550718:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c562f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 73 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550722:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c562f lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 73 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550725:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.550727:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c562f lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 73 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550730:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c400/0x4605c7c97e1c562f lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 73 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550736:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c5636 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 72 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550739:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c5636 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 72 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550742:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c5636 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 72 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550745:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c5636 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 72 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550837:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c5636 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 72 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550843:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839ca00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.550845:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c5636 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 72 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550848:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c5636 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 72 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550853:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c5636 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 72 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550856:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.550859:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c5636 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 72 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550862:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839ca00/0x4605c7c97e1c5636 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 72 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550871:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c563d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 71 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550914:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c563d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 71 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550920:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c563d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 71 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550926:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c563d lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 71 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550932:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c563d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 71 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550937:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839da00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.550941:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c563d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 71 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550946:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c563d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 71 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550951:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c563d lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 71 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550954:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.550956:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c563d lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 71 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550960:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839da00/0x4605c7c97e1c563d lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 71 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550969:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c5644 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 70 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550973:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c5644 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 70 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550977:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c5644 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 70 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550980:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c5644 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 70 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550983:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c5644 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 70 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.550985:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839c000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.550987:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c5644 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 70 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551263:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c5644 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 70 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551277:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c5644 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 70 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551282:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.551286:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c5644 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 70 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551292:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c000/0x4605c7c97e1c5644 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 70 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551306:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c564b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 69 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551312:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c564b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 69 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551317:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c564b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 69 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551322:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c564b lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 69 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551327:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c564b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 69 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551332:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839c600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.551335:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c564b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 69 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551340:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c564b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 69 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551347:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c564b lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 69 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551352:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.551355:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c564b lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 69 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551360:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c600/0x4605c7c97e1c564b lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 69 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551372:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c5652 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 68 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551379:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c5652 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 68 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551384:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c5652 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 68 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551389:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c5652 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 68 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551394:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c5652 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 68 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551399:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839c800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.551402:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c5652 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 68 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551407:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c5652 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 68 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551414:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c5652 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 68 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551420:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.551423:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c5652 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 68 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551428:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839c800/0x4605c7c97e1c5652 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 68 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.551451:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902345831180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442745984:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:1.0:1788201389.551530:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5819 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 67 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551537:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5819 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 67 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551542:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5819 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 67 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551548:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5819 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 67 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551554:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5819 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 67 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551559:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865ac00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.551561:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5819 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 67 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551571:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5819 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 67 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551576:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5819 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 67 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551581:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.551584:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5819 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 67 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.551590:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ac00/0x4605c7c97e1c5819 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 67 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.551665:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442746496:192.168.201.130@tcp:63:ll_sa.0 00000100:00100000:0.0:1788201389.553538:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442746240:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201389.553775:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201389.553787:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201389.553840:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.553853:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x41d:0x0](ffff9023449a00c8) 00000008:00010000:0.0:1788201389.553920:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.554690:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.554712:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.554729:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00010000:00010000:0.0:1788201389.554744:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00000080:00200000:0.0:1788201389.554753:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.554760:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.554763:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.554771:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:0.0:1788201389.554774:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00010000:00010000:0.0:1788201389.554783:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00000080:00200000:0.0:1788201389.554800:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.554807:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.554812:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.554816:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-19, flags=8 00010000:00010000:0.0:1788201389.554827:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1788201389.554834:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53f8 from it ffffa0b181c3baf8 00010000:00010000:0.0:1788201389.554837:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.554843:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:0.0:1788201389.554863:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-19:[0x200000bd1:0x41e:0x0](ffff9023449a0908), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201389.554869:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x41e:0x0](ffff9023449a0908),name=t-19 00000002:00010000:0.0:1788201389.554901:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x41e:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.554909:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.554918:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201389.554924:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.554928:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x41e:0x0](ffff9023449a0908) 00000080:00010000:0.0:1788201389.554931:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x41e:0x0](ffff9023449a0908) for lock 0x4605c7c97e1c53f8 00000080:00010000:0.0:1788201389.554935:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53f8 from it ffffa0b181c3bcb8 00010000:00010000:0.0:1788201389.554938:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.554946:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.555144:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.555151:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x41e:0x0] 00000008:00010000:0.0:1788201389.555299:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201389.555303:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902350050eb0, osc lock ffff902343fb7db0, flags 0x400001002 00010000:00010000:0.0:1788201389.555323:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.555339:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1788201389.555480:0:17859:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1788201389.555490:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788201389.555494:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.555502:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1788201389.555513:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788201389.556300:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x41e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201389.556308:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.556312:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.556316:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350050eb0 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.556320:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.556323:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350050eb0 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.556488:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:0.0:1788201389.561822:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5659 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 66 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561839:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5659 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 66 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561848:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5659 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 66 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561855:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5659 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 66 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561862:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5659 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 66 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561866:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839d000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.561869:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5659 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 66 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561874:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5659 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 66 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561892:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5659 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 66 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561897:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.561902:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5659 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 66 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561906:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d000/0x4605c7c97e1c5659 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 66 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561926:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c5660 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 65 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561929:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c5660 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 65 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561934:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c5660 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 65 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561937:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c5660 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 65 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561940:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c5660 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 65 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561943:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839d800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.561945:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c5660 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 65 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561948:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c5660 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 65 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561951:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c5660 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 65 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561953:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.561955:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c5660 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 65 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561958:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d800/0x4605c7c97e1c5660 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 65 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561965:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5667 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 64 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561968:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5667 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 64 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561972:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5667 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 64 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561976:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5667 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 64 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561979:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5667 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 64 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561981:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902343730400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.561983:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5667 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 64 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561986:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5667 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 64 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561989:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5667 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 64 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.561994:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.561996:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5667 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 64 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562001:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730400/0x4605c7c97e1c5667 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 64 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562011:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c566e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 63 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562016:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c566e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 63 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562022:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c566e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 63 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562027:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c566e lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 63 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562033:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c566e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 63 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562038:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902343731200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.562041:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c566e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 63 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562046:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c566e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 63 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562051:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c566e lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 63 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562056:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.562059:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c566e lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 63 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562064:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731200/0x4605c7c97e1c566e lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 63 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562075:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124c00/0x4605c7c97e1c5675 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 62 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562080:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124c00/0x4605c7c97e1c5675 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 62 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562084:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124c00/0x4605c7c97e1c5675 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 62 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562087:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124c00/0x4605c7c97e1c5675 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 62 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562090:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124c00/0x4605c7c97e1c5675 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 62 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562093:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902353124c00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.562095:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124c00/0x4605c7c97e1c5675 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 62 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562100:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124c00/0x4605c7c97e1c5675 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 62 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562105:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124c00/0x4605c7c97e1c5675 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 62 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562109:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.562112:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124c00/0x4605c7c97e1c5675 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 62 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562117:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124c00/0x4605c7c97e1c5675 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 62 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562128:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125c00/0x4605c7c97e1c567c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 61 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562134:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125c00/0x4605c7c97e1c567c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 61 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562140:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125c00/0x4605c7c97e1c567c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 61 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562145:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125c00/0x4605c7c97e1c567c lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 61 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562150:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125c00/0x4605c7c97e1c567c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 61 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562155:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902353125c00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.562170:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125c00/0x4605c7c97e1c567c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 61 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562176:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125c00/0x4605c7c97e1c567c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 61 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562183:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125c00/0x4605c7c97e1c567c lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 61 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562188:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.562192:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125c00/0x4605c7c97e1c567c lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 61 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562197:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125c00/0x4605c7c97e1c567c lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 61 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562209:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125000/0x4605c7c97e1c5683 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 60 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562215:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125000/0x4605c7c97e1c5683 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 60 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562220:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125000/0x4605c7c97e1c5683 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 60 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562224:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125000/0x4605c7c97e1c5683 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 60 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562227:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125000/0x4605c7c97e1c5683 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 60 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562230:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902353125000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.562231:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125000/0x4605c7c97e1c5683 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 60 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562234:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125000/0x4605c7c97e1c5683 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 60 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562238:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125000/0x4605c7c97e1c5683 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 60 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562241:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.562243:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125000/0x4605c7c97e1c5683 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 60 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562246:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125000/0x4605c7c97e1c5683 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 60 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562253:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125600/0x4605c7c97e1c568a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 59 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562256:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125600/0x4605c7c97e1c568a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 59 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562260:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125600/0x4605c7c97e1c568a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 59 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562264:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125600/0x4605c7c97e1c568a lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 59 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562277:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125600/0x4605c7c97e1c568a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 59 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562282:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902353125600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.562284:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125600/0x4605c7c97e1c568a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 59 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562290:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125600/0x4605c7c97e1c568a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 59 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562296:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125600/0x4605c7c97e1c568a lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 59 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562302:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.562305:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125600/0x4605c7c97e1c568a lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 59 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562310:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125600/0x4605c7c97e1c568a lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 59 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562322:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124600/0x4605c7c97e1c5691 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 58 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562328:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124600/0x4605c7c97e1c5691 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 58 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562334:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124600/0x4605c7c97e1c5691 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 58 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562789:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124600/0x4605c7c97e1c5691 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 58 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562800:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124600/0x4605c7c97e1c5691 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 58 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562805:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902353124600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.562808:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124600/0x4605c7c97e1c5691 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 58 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562814:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124600/0x4605c7c97e1c5691 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 58 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562820:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124600/0x4605c7c97e1c5691 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 58 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562825:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.562829:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124600/0x4605c7c97e1c5691 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 58 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562834:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124600/0x4605c7c97e1c5691 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 58 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562849:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125e00/0x4605c7c97e1c5698 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 57 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562856:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125e00/0x4605c7c97e1c5698 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 57 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562862:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125e00/0x4605c7c97e1c5698 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 57 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562868:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125e00/0x4605c7c97e1c5698 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 57 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562873:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125e00/0x4605c7c97e1c5698 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 57 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562904:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902353125e00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.562906:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125e00/0x4605c7c97e1c5698 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 57 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562912:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125e00/0x4605c7c97e1c5698 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 57 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562917:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125e00/0x4605c7c97e1c5698 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 57 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562923:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.562926:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125e00/0x4605c7c97e1c5698 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 57 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562931:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125e00/0x4605c7c97e1c5698 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 57 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562943:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125800/0x4605c7c97e1c569f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 56 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562949:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125800/0x4605c7c97e1c569f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 56 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562954:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125800/0x4605c7c97e1c569f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 56 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562959:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125800/0x4605c7c97e1c569f lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 56 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562965:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125800/0x4605c7c97e1c569f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 56 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562970:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902353125800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.562973:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125800/0x4605c7c97e1c569f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 56 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562978:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125800/0x4605c7c97e1c569f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 56 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562983:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125800/0x4605c7c97e1c569f lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 56 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562988:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.562991:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125800/0x4605c7c97e1c569f lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 56 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.562997:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125800/0x4605c7c97e1c569f lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 56 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563009:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124400/0x4605c7c97e1c56a6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 55 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563015:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124400/0x4605c7c97e1c56a6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 55 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563021:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124400/0x4605c7c97e1c56a6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 55 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563027:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124400/0x4605c7c97e1c56a6 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 55 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563033:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124400/0x4605c7c97e1c56a6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 55 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563038:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902353124400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.563040:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124400/0x4605c7c97e1c56a6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 55 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563046:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124400/0x4605c7c97e1c56a6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 55 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563052:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124400/0x4605c7c97e1c56a6 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 55 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563057:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.563060:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124400/0x4605c7c97e1c56a6 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 55 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563066:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353124400/0x4605c7c97e1c56a6 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 55 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563077:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125400/0x4605c7c97e1c56ad lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 54 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563083:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125400/0x4605c7c97e1c56ad lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 54 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563089:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125400/0x4605c7c97e1c56ad lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 54 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563094:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125400/0x4605c7c97e1c56ad lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 54 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563099:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125400/0x4605c7c97e1c56ad lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 54 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563104:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902353125400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.563107:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125400/0x4605c7c97e1c56ad lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 54 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563112:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125400/0x4605c7c97e1c56ad lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 54 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563116:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125400/0x4605c7c97e1c56ad lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 54 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563121:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.563124:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125400/0x4605c7c97e1c56ad lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 54 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563129:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125400/0x4605c7c97e1c56ad lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 54 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563140:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125200/0x4605c7c97e1c56b4 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 53 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563146:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125200/0x4605c7c97e1c56b4 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 53 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563152:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125200/0x4605c7c97e1c56b4 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 53 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563402:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125200/0x4605c7c97e1c56b4 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 53 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563411:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125200/0x4605c7c97e1c56b4 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 53 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563417:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902353125200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.563420:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125200/0x4605c7c97e1c56b4 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 53 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563426:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125200/0x4605c7c97e1c56b4 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 53 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563432:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125200/0x4605c7c97e1c56b4 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 53 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563437:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.563440:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125200/0x4605c7c97e1c56b4 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 53 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563446:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125200/0x4605c7c97e1c56b4 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 53 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563458:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125a00/0x4605c7c97e1c56bb lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 52 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563463:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125a00/0x4605c7c97e1c56bb lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 52 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563469:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125a00/0x4605c7c97e1c56bb lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 52 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563475:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125a00/0x4605c7c97e1c56bb lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 52 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563481:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125a00/0x4605c7c97e1c56bb lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 52 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563486:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902353125a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.563489:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125a00/0x4605c7c97e1c56bb lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 52 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563494:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125a00/0x4605c7c97e1c56bb lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 52 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563499:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125a00/0x4605c7c97e1c56bb lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 52 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563504:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.563507:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125a00/0x4605c7c97e1c56bb lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 52 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563513:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902353125a00/0x4605c7c97e1c56bb lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 52 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563523:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64e00/0x4605c7c97e1c56c2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 51 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563692:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64e00/0x4605c7c97e1c56c2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 51 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563698:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64e00/0x4605c7c97e1c56c2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 51 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563703:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64e00/0x4605c7c97e1c56c2 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 51 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563709:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64e00/0x4605c7c97e1c56c2 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 51 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563714:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352e64e00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.563716:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64e00/0x4605c7c97e1c56c2 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 51 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563722:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64e00/0x4605c7c97e1c56c2 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 51 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563727:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64e00/0x4605c7c97e1c56c2 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 51 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563731:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.563734:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64e00/0x4605c7c97e1c56c2 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 51 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563740:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64e00/0x4605c7c97e1c56c2 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 51 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563752:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65200/0x4605c7c97e1c56c9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 50 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563757:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65200/0x4605c7c97e1c56c9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 50 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563762:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65200/0x4605c7c97e1c56c9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 50 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563768:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65200/0x4605c7c97e1c56c9 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 50 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563773:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65200/0x4605c7c97e1c56c9 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 50 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563778:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352e65200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.563780:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65200/0x4605c7c97e1c56c9 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 50 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563785:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65200/0x4605c7c97e1c56c9 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 50 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563790:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65200/0x4605c7c97e1c56c9 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 50 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563795:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.563797:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65200/0x4605c7c97e1c56c9 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 50 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563802:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65200/0x4605c7c97e1c56c9 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 50 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563814:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65c00/0x4605c7c97e1c56d0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 49 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563820:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65c00/0x4605c7c97e1c56d0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 49 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563825:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65c00/0x4605c7c97e1c56d0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 49 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563830:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65c00/0x4605c7c97e1c56d0 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 49 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563835:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65c00/0x4605c7c97e1c56d0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 49 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563840:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352e65c00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.563842:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65c00/0x4605c7c97e1c56d0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 49 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563848:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65c00/0x4605c7c97e1c56d0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 49 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563853:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65c00/0x4605c7c97e1c56d0 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 49 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563857:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.563859:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65c00/0x4605c7c97e1c56d0 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 49 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563864:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65c00/0x4605c7c97e1c56d0 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 49 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563874:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64800/0x4605c7c97e1c56d7 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 48 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563904:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64800/0x4605c7c97e1c56d7 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 48 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563911:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64800/0x4605c7c97e1c56d7 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 48 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563917:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64800/0x4605c7c97e1c56d7 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 48 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563922:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64800/0x4605c7c97e1c56d7 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 48 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563927:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352e64800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.563929:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64800/0x4605c7c97e1c56d7 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 48 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563935:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64800/0x4605c7c97e1c56d7 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 48 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563941:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64800/0x4605c7c97e1c56d7 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 48 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563946:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.563949:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64800/0x4605c7c97e1c56d7 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 48 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563954:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64800/0x4605c7c97e1c56d7 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 48 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563966:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65800/0x4605c7c97e1c56de lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 47 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563972:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65800/0x4605c7c97e1c56de lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 47 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563978:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65800/0x4605c7c97e1c56de lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 47 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563983:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65800/0x4605c7c97e1c56de lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 47 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563989:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65800/0x4605c7c97e1c56de lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 47 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.563993:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352e65800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.563996:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65800/0x4605c7c97e1c56de lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 47 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564002:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65800/0x4605c7c97e1c56de lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 47 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564008:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65800/0x4605c7c97e1c56de lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 47 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564014:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.564017:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65800/0x4605c7c97e1c56de lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 47 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564023:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65800/0x4605c7c97e1c56de lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 47 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564035:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65a00/0x4605c7c97e1c56e5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 46 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564040:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65a00/0x4605c7c97e1c56e5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 46 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564046:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65a00/0x4605c7c97e1c56e5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 46 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564052:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65a00/0x4605c7c97e1c56e5 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 46 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564057:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65a00/0x4605c7c97e1c56e5 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 46 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564124:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352e65a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.564128:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65a00/0x4605c7c97e1c56e5 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 46 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564134:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65a00/0x4605c7c97e1c56e5 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 46 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564140:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65a00/0x4605c7c97e1c56e5 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 46 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564146:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.564150:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65a00/0x4605c7c97e1c56e5 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 46 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564278:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65a00/0x4605c7c97e1c56e5 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 46 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564292:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65600/0x4605c7c97e1c56ec lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 45 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564297:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65600/0x4605c7c97e1c56ec lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 45 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564302:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65600/0x4605c7c97e1c56ec lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 45 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564307:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65600/0x4605c7c97e1c56ec lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 45 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564312:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65600/0x4605c7c97e1c56ec lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 45 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564316:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352e65600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.564319:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65600/0x4605c7c97e1c56ec lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 45 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564323:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65600/0x4605c7c97e1c56ec lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 45 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564328:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65600/0x4605c7c97e1c56ec lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 45 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564333:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000080:00200000:1.0:1788201389.564334:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00010000:00010000:0.0:1788201389.564336:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65600/0x4605c7c97e1c56ec lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 45 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201389.564355:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.564372:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00010000:00010000:1.0:1788201389.564385:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00000080:00200000:1.0:1788201389.564393:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.564399:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.564403:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00010000:00010000:0.0:1788201389.564407:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65600/0x4605c7c97e1c56ec lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 45 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00010000:1.0:1788201389.564408:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:1.0:1788201389.564412:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00010000:00010000:0.0:1788201389.564419:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65e00/0x4605c7c97e1c56f3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 44 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564426:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65e00/0x4605c7c97e1c56f3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 44 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.564426:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00010000:00010000:0.0:1788201389.564433:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65e00/0x4605c7c97e1c56f3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 44 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564437:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65e00/0x4605c7c97e1c56f3 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 44 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564442:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65e00/0x4605c7c97e1c56f3 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 44 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1788201389.564442:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00010000:00010000:0.0:1788201389.564447:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352e65e00 already unused, calling callback (ffffffffc1a662a0) 00000080:00200000:1.0:1788201389.564448:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00010000:00010000:0.0:1788201389.564450:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65e00/0x4605c7c97e1c56f3 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 44 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1788201389.564453:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00010000:00010000:0.0:1788201389.564455:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65e00/0x4605c7c97e1c56f3 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 44 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1788201389.564460:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-19, flags=0 00010000:00010000:0.0:1788201389.564461:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65e00/0x4605c7c97e1c56f3 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 44 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1788201389.564464:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00010000:00010000:0.0:1788201389.564466:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000080:00200000:1.0:1788201389.564469:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-19, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:0.0:1788201389.564470:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65e00/0x4605c7c97e1c56f3 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 44 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564475:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e65e00/0x4605c7c97e1c56f3 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 44 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564484:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64000/0x4605c7c97e1c56fa lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 43 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564489:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64000/0x4605c7c97e1c56fa lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 43 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564494:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64000/0x4605c7c97e1c56fa lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 43 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564499:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64000/0x4605c7c97e1c56fa lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 43 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564504:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64000/0x4605c7c97e1c56fa lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 43 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564509:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352e64000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.564511:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64000/0x4605c7c97e1c56fa lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 43 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564516:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64000/0x4605c7c97e1c56fa lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 43 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564521:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64000/0x4605c7c97e1c56fa lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 43 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.564548:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201389.564569:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x41e:0x0]: losing layout lock 00010000:00010000:1.0:1788201389.564615:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201389.564622:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201389.564625:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802200/0x4605c7c97e1c53f8 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x41e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d55e9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:1.0:1788201389.564648:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201389.564662:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234519c700 x1875064442746752/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1788201389.564683:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.564687:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64000/0x4605c7c97e1c56fa lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 43 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.564690:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442746752:192.168.201.130@tcp:36:rm.0 00010000:00010000:0.0:1788201389.564693:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64000/0x4605c7c97e1c56fa lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 43 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564705:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64200/0x4605c7c97e1c5701 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 42 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564710:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64200/0x4605c7c97e1c5701 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 42 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564715:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64200/0x4605c7c97e1c5701 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 42 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564720:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64200/0x4605c7c97e1c5701 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 42 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564725:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64200/0x4605c7c97e1c5701 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 42 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564729:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352e64200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.564731:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64200/0x4605c7c97e1c5701 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 42 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564736:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64200/0x4605c7c97e1c5701 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 42 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564741:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64200/0x4605c7c97e1c5701 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 42 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564746:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.564749:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64200/0x4605c7c97e1c5701 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 42 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564753:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64200/0x4605c7c97e1c5701 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 42 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564763:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64a00/0x4605c7c97e1c5708 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 41 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564768:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64a00/0x4605c7c97e1c5708 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 41 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564773:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64a00/0x4605c7c97e1c5708 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 41 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.564777:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff902345676c00 going to sleep for 11 seconds 00010000:00010000:0.0:1788201389.564778:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64a00/0x4605c7c97e1c5708 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 41 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564783:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64a00/0x4605c7c97e1c5708 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 41 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564788:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352e64a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.564790:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64a00/0x4605c7c97e1c5708 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 41 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564795:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64a00/0x4605c7c97e1c5708 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 41 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564800:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64a00/0x4605c7c97e1c5708 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 41 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564805:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.564808:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64a00/0x4605c7c97e1c5708 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 41 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564814:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352e64a00/0x4605c7c97e1c5708 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 41 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564825:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c000/0x4605c7c97e1c570f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 40 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564830:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c000/0x4605c7c97e1c570f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 40 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564835:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c000/0x4605c7c97e1c570f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 40 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564839:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c000/0x4605c7c97e1c570f lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 40 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564845:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c000/0x4605c7c97e1c570f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 40 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564849:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235869c000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.564852:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c000/0x4605c7c97e1c570f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 40 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564857:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c000/0x4605c7c97e1c570f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 40 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564862:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c000/0x4605c7c97e1c570f lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 40 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564867:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.564870:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c000/0x4605c7c97e1c570f lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 40 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564898:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c000/0x4605c7c97e1c570f lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 40 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564910:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d200/0x4605c7c97e1c5716 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 39 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564915:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d200/0x4605c7c97e1c5716 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 39 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564921:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d200/0x4605c7c97e1c5716 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 39 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564926:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d200/0x4605c7c97e1c5716 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 39 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564932:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d200/0x4605c7c97e1c5716 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 39 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564937:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235869d200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.564940:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d200/0x4605c7c97e1c5716 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 39 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564945:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d200/0x4605c7c97e1c5716 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 39 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564951:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d200/0x4605c7c97e1c5716 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 39 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564956:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.564959:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d200/0x4605c7c97e1c5716 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 39 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564965:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d200/0x4605c7c97e1c5716 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 39 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564976:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869dc00/0x4605c7c97e1c571d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 38 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564981:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869dc00/0x4605c7c97e1c571d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 38 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564987:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869dc00/0x4605c7c97e1c571d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 38 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564992:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869dc00/0x4605c7c97e1c571d lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 38 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.564998:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869dc00/0x4605c7c97e1c571d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 38 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565002:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235869dc00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.565005:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869dc00/0x4605c7c97e1c571d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 38 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565011:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869dc00/0x4605c7c97e1c571d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 38 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565017:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869dc00/0x4605c7c97e1c571d lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 38 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565022:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.565025:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869dc00/0x4605c7c97e1c571d lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 38 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565031:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869dc00/0x4605c7c97e1c571d lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 38 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565041:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869da00/0x4605c7c97e1c5724 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 37 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565047:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869da00/0x4605c7c97e1c5724 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 37 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565053:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869da00/0x4605c7c97e1c5724 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 37 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565259:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869da00/0x4605c7c97e1c5724 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 37 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565278:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869da00/0x4605c7c97e1c5724 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 37 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565284:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235869da00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.565286:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869da00/0x4605c7c97e1c5724 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 37 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565292:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869da00/0x4605c7c97e1c5724 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 37 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565298:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869da00/0x4605c7c97e1c5724 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 37 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565304:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.565308:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869da00/0x4605c7c97e1c5724 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 37 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565314:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869da00/0x4605c7c97e1c5724 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 37 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565327:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c200/0x4605c7c97e1c572b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 36 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565332:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c200/0x4605c7c97e1c572b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 36 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565338:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c200/0x4605c7c97e1c572b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 36 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565412:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c200/0x4605c7c97e1c572b lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 36 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565418:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c200/0x4605c7c97e1c572b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 36 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565423:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235869c200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.565425:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c200/0x4605c7c97e1c572b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 36 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565430:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c200/0x4605c7c97e1c572b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 36 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565436:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c200/0x4605c7c97e1c572b lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 36 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565442:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.565445:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c200/0x4605c7c97e1c572b lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 36 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565450:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c200/0x4605c7c97e1c572b lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 36 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565463:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869de00/0x4605c7c97e1c5732 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 35 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565469:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869de00/0x4605c7c97e1c5732 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 35 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565476:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869de00/0x4605c7c97e1c5732 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 35 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565481:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869de00/0x4605c7c97e1c5732 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 35 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565487:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869de00/0x4605c7c97e1c5732 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 35 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565491:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235869de00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.565494:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869de00/0x4605c7c97e1c5732 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 35 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565500:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869de00/0x4605c7c97e1c5732 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 35 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565506:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869de00/0x4605c7c97e1c5732 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 35 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565511:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.565514:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869de00/0x4605c7c97e1c5732 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 35 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565519:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869de00/0x4605c7c97e1c5732 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 35 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565701:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d600/0x4605c7c97e1c5739 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 34 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565707:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d600/0x4605c7c97e1c5739 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 34 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565713:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d600/0x4605c7c97e1c5739 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 34 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565719:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d600/0x4605c7c97e1c5739 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 34 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565725:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d600/0x4605c7c97e1c5739 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 34 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565730:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235869d600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.565733:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d600/0x4605c7c97e1c5739 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 34 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565738:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d600/0x4605c7c97e1c5739 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 34 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565744:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d600/0x4605c7c97e1c5739 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 34 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565749:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.565753:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d600/0x4605c7c97e1c5739 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 34 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565759:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d600/0x4605c7c97e1c5739 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 34 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565769:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c800/0x4605c7c97e1c5740 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 33 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565775:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c800/0x4605c7c97e1c5740 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 33 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565781:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c800/0x4605c7c97e1c5740 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 33 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565787:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c800/0x4605c7c97e1c5740 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 33 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565793:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c800/0x4605c7c97e1c5740 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 33 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565797:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235869c800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.565800:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c800/0x4605c7c97e1c5740 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 33 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565805:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c800/0x4605c7c97e1c5740 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 33 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565810:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c800/0x4605c7c97e1c5740 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 33 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565815:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.565818:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c800/0x4605c7c97e1c5740 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 33 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565823:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c800/0x4605c7c97e1c5740 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 33 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565832:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869cc00/0x4605c7c97e1c5747 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 32 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565838:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869cc00/0x4605c7c97e1c5747 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 32 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565844:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869cc00/0x4605c7c97e1c5747 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 32 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565850:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869cc00/0x4605c7c97e1c5747 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 32 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565855:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869cc00/0x4605c7c97e1c5747 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 32 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565860:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235869cc00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.565863:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869cc00/0x4605c7c97e1c5747 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 32 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565868:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869cc00/0x4605c7c97e1c5747 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 32 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565873:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869cc00/0x4605c7c97e1c5747 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 32 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565907:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.565910:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869cc00/0x4605c7c97e1c5747 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 32 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565916:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869cc00/0x4605c7c97e1c5747 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 32 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565925:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d800/0x4605c7c97e1c574e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 31 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565931:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d800/0x4605c7c97e1c574e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 31 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565937:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d800/0x4605c7c97e1c574e lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 31 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565943:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d800/0x4605c7c97e1c574e lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 31 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565948:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d800/0x4605c7c97e1c574e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 31 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565953:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235869d800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.565956:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d800/0x4605c7c97e1c574e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 31 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565961:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d800/0x4605c7c97e1c574e lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 31 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565966:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d800/0x4605c7c97e1c574e lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 31 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565971:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.565974:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d800/0x4605c7c97e1c574e lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 31 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565979:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869d800/0x4605c7c97e1c574e lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 31 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565989:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c600/0x4605c7c97e1c5755 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 30 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.565994:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c600/0x4605c7c97e1c5755 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 30 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566000:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c600/0x4605c7c97e1c5755 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 30 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566006:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c600/0x4605c7c97e1c5755 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 30 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566012:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c600/0x4605c7c97e1c5755 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 30 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566018:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235869c600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.566021:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c600/0x4605c7c97e1c5755 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 30 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566027:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c600/0x4605c7c97e1c5755 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 30 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566032:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c600/0x4605c7c97e1c5755 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 30 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566037:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.566040:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c600/0x4605c7c97e1c5755 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 30 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566046:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869c600/0x4605c7c97e1c5755 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 30 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566056:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869ce00/0x4605c7c97e1c575c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 29 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566128:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869ce00/0x4605c7c97e1c575c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 29 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566134:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869ce00/0x4605c7c97e1c575c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 29 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566141:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869ce00/0x4605c7c97e1c575c lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 29 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566147:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869ce00/0x4605c7c97e1c575c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 29 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566153:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235869ce00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.566258:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869ce00/0x4605c7c97e1c575c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 29 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566275:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869ce00/0x4605c7c97e1c575c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 29 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566283:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869ce00/0x4605c7c97e1c575c lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 29 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566288:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.566292:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869ce00/0x4605c7c97e1c575c lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 29 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566299:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235869ce00/0x4605c7c97e1c575c lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 29 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566312:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733000/0x4605c7c97e1c5763 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 28 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566317:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733000/0x4605c7c97e1c5763 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 28 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566324:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733000/0x4605c7c97e1c5763 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 28 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566330:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733000/0x4605c7c97e1c5763 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 28 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566336:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733000/0x4605c7c97e1c5763 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 28 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566400:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352733000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.566404:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733000/0x4605c7c97e1c5763 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 28 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566410:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733000/0x4605c7c97e1c5763 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 28 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566415:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733000/0x4605c7c97e1c5763 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 28 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566421:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.566424:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733000/0x4605c7c97e1c5763 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 28 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566430:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733000/0x4605c7c97e1c5763 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 28 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566441:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732a00/0x4605c7c97e1c576a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 27 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566447:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732a00/0x4605c7c97e1c576a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 27 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566453:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732a00/0x4605c7c97e1c576a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 27 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566459:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732a00/0x4605c7c97e1c576a lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 27 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566465:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732a00/0x4605c7c97e1c576a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 27 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566470:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352732a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.566472:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732a00/0x4605c7c97e1c576a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 27 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566477:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732a00/0x4605c7c97e1c576a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 27 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566483:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732a00/0x4605c7c97e1c576a lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 27 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566488:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.566490:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732a00/0x4605c7c97e1c576a lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 27 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566495:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732a00/0x4605c7c97e1c576a lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 27 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566504:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733a00/0x4605c7c97e1c5771 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 26 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566510:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733a00/0x4605c7c97e1c5771 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 26 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566516:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733a00/0x4605c7c97e1c5771 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 26 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566521:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733a00/0x4605c7c97e1c5771 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 26 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566676:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733a00/0x4605c7c97e1c5771 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 26 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566681:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352733a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.566684:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733a00/0x4605c7c97e1c5771 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 26 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566689:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733a00/0x4605c7c97e1c5771 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 26 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566695:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733a00/0x4605c7c97e1c5771 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 26 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566700:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.566704:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733a00/0x4605c7c97e1c5771 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 26 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566710:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733a00/0x4605c7c97e1c5771 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 26 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566722:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732600/0x4605c7c97e1c5778 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566728:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732600/0x4605c7c97e1c5778 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566735:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732600/0x4605c7c97e1c5778 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566740:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732600/0x4605c7c97e1c5778 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566746:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732600/0x4605c7c97e1c5778 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 25 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566750:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352732600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.566753:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732600/0x4605c7c97e1c5778 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 25 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566758:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732600/0x4605c7c97e1c5778 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 25 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566763:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732600/0x4605c7c97e1c5778 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 25 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566768:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.566771:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732600/0x4605c7c97e1c5778 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 25 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566776:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732600/0x4605c7c97e1c5778 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 25 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566786:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732400/0x4605c7c97e1c577f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566791:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732400/0x4605c7c97e1c577f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566798:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732400/0x4605c7c97e1c577f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566803:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732400/0x4605c7c97e1c577f lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566809:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732400/0x4605c7c97e1c577f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 24 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566813:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352732400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.566816:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732400/0x4605c7c97e1c577f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 24 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566821:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732400/0x4605c7c97e1c577f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 24 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566827:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732400/0x4605c7c97e1c577f lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 24 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566832:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.566834:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732400/0x4605c7c97e1c577f lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 24 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566840:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732400/0x4605c7c97e1c577f lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 24 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566849:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733600/0x4605c7c97e1c5786 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 23 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566856:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733600/0x4605c7c97e1c5786 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 23 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566863:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733600/0x4605c7c97e1c5786 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 23 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566869:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733600/0x4605c7c97e1c5786 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 23 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566897:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733600/0x4605c7c97e1c5786 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 23 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566902:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352733600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.566905:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733600/0x4605c7c97e1c5786 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 23 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566911:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733600/0x4605c7c97e1c5786 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 23 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566917:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733600/0x4605c7c97e1c5786 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 23 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.566922:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.566925:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733600/0x4605c7c97e1c5786 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 23 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567047:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733600/0x4605c7c97e1c5786 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 23 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567120:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733e00/0x4605c7c97e1c578d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567128:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733e00/0x4605c7c97e1c578d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567134:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733e00/0x4605c7c97e1c578d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567140:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733e00/0x4605c7c97e1c578d lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567146:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733e00/0x4605c7c97e1c578d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 22 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567151:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352733e00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.567154:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733e00/0x4605c7c97e1c578d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 22 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567264:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733e00/0x4605c7c97e1c578d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 22 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567280:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733e00/0x4605c7c97e1c578d lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 22 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567286:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.567290:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733e00/0x4605c7c97e1c578d lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 22 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567296:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733e00/0x4605c7c97e1c578d lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 22 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567309:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733800/0x4605c7c97e1c5794 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567315:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733800/0x4605c7c97e1c5794 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567322:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733800/0x4605c7c97e1c5794 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567328:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733800/0x4605c7c97e1c5794 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567333:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733800/0x4605c7c97e1c5794 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 21 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567338:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352733800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.567399:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733800/0x4605c7c97e1c5794 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 21 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567406:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733800/0x4605c7c97e1c5794 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 21 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567412:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733800/0x4605c7c97e1c5794 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 21 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567417:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.567420:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733800/0x4605c7c97e1c5794 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 21 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567426:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733800/0x4605c7c97e1c5794 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 21 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567438:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732800/0x4605c7c97e1c579b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567444:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732800/0x4605c7c97e1c579b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567451:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732800/0x4605c7c97e1c579b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567457:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732800/0x4605c7c97e1c579b lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567462:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732800/0x4605c7c97e1c579b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 20 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567467:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352732800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.567469:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732800/0x4605c7c97e1c579b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 20 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567475:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732800/0x4605c7c97e1c579b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 20 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567480:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732800/0x4605c7c97e1c579b lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 20 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567485:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.567487:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732800/0x4605c7c97e1c579b lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 20 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567493:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732800/0x4605c7c97e1c579b lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 20 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567502:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732e00/0x4605c7c97e1c57a2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 19 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567508:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732e00/0x4605c7c97e1c57a2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 19 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567514:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732e00/0x4605c7c97e1c57a2 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 19 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567520:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732e00/0x4605c7c97e1c57a2 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 19 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567534:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732e00/0x4605c7c97e1c57a2 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 19 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567538:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352732e00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.567541:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732e00/0x4605c7c97e1c57a2 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 19 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567672:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732e00/0x4605c7c97e1c57a2 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 19 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567679:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732e00/0x4605c7c97e1c57a2 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 19 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567685:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.567688:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732e00/0x4605c7c97e1c57a2 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 19 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567694:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732e00/0x4605c7c97e1c57a2 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 19 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567707:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733400/0x4605c7c97e1c57a9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567714:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733400/0x4605c7c97e1c57a9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567720:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733400/0x4605c7c97e1c57a9 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567726:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733400/0x4605c7c97e1c57a9 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567731:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733400/0x4605c7c97e1c57a9 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 18 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567736:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352733400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.567739:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733400/0x4605c7c97e1c57a9 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 18 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567744:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733400/0x4605c7c97e1c57a9 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 18 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567749:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733400/0x4605c7c97e1c57a9 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 18 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567754:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.567757:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733400/0x4605c7c97e1c57a9 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 18 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567762:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352733400/0x4605c7c97e1c57a9 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 18 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567772:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732000/0x4605c7c97e1c57b0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567778:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732000/0x4605c7c97e1c57b0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567784:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732000/0x4605c7c97e1c57b0 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567790:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732000/0x4605c7c97e1c57b0 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567795:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732000/0x4605c7c97e1c57b0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 17 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567800:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902352732000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.567802:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732000/0x4605c7c97e1c57b0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 17 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567807:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732000/0x4605c7c97e1c57b0 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 17 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567813:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732000/0x4605c7c97e1c57b0 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 17 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567818:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.567820:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732000/0x4605c7c97e1c57b0 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 17 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567826:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902352732000/0x4605c7c97e1c57b0 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 17 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567836:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756c00/0x4605c7c97e1c57b7 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567841:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756c00/0x4605c7c97e1c57b7 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567848:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756c00/0x4605c7c97e1c57b7 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567853:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756c00/0x4605c7c97e1c57b7 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567859:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756c00/0x4605c7c97e1c57b7 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 16 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567863:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902358756c00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.567866:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756c00/0x4605c7c97e1c57b7 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 16 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567871:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756c00/0x4605c7c97e1c57b7 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 16 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567901:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756c00/0x4605c7c97e1c57b7 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 16 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567906:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.567909:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756c00/0x4605c7c97e1c57b7 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 16 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567914:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756c00/0x4605c7c97e1c57b7 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 16 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567924:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757c00/0x4605c7c97e1c57be lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567930:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757c00/0x4605c7c97e1c57be lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567936:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757c00/0x4605c7c97e1c57be lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567942:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757c00/0x4605c7c97e1c57be lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567948:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757c00/0x4605c7c97e1c57be lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 15 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567953:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902358757c00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.567955:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757c00/0x4605c7c97e1c57be lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 15 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567961:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757c00/0x4605c7c97e1c57be lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 15 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567966:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757c00/0x4605c7c97e1c57be lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 15 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567970:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.567973:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757c00/0x4605c7c97e1c57be lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 15 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567978:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757c00/0x4605c7c97e1c57be lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 15 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567987:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757200/0x4605c7c97e1c57c5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567992:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757200/0x4605c7c97e1c57c5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.567998:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757200/0x4605c7c97e1c57c5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568003:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757200/0x4605c7c97e1c57c5 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568009:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757200/0x4605c7c97e1c57c5 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 14 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568015:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902358757200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.568018:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757200/0x4605c7c97e1c57c5 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 14 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568024:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757200/0x4605c7c97e1c57c5 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 14 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568030:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757200/0x4605c7c97e1c57c5 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 14 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568035:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.568038:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757200/0x4605c7c97e1c57c5 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 14 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568043:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757200/0x4605c7c97e1c57c5 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 14 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568053:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757800/0x4605c7c97e1c57cc lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568070:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757800/0x4605c7c97e1c57cc lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568077:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757800/0x4605c7c97e1c57cc lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568084:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757800/0x4605c7c97e1c57cc lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568090:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757800/0x4605c7c97e1c57cc lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 13 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568095:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902358757800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.568097:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757800/0x4605c7c97e1c57cc lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568103:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757800/0x4605c7c97e1c57cc lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568108:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757800/0x4605c7c97e1c57cc lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568114:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.568116:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757800/0x4605c7c97e1c57cc lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568122:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757800/0x4605c7c97e1c57cc lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 13 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568132:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757a00/0x4605c7c97e1c57d3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568137:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757a00/0x4605c7c97e1c57d3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568143:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757a00/0x4605c7c97e1c57d3 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568149:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757a00/0x4605c7c97e1c57d3 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568154:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757a00/0x4605c7c97e1c57d3 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568167:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902358757a00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.568169:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757a00/0x4605c7c97e1c57d3 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 12 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568174:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757a00/0x4605c7c97e1c57d3 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 12 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568180:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757a00/0x4605c7c97e1c57d3 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 12 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568186:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.568189:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757a00/0x4605c7c97e1c57d3 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 12 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568195:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757a00/0x4605c7c97e1c57d3 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 12 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568205:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756e00/0x4605c7c97e1c57da lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568210:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756e00/0x4605c7c97e1c57da lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568216:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756e00/0x4605c7c97e1c57da lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568222:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756e00/0x4605c7c97e1c57da lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568228:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756e00/0x4605c7c97e1c57da lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568232:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902358756e00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.568235:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756e00/0x4605c7c97e1c57da lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568240:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756e00/0x4605c7c97e1c57da lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568245:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756e00/0x4605c7c97e1c57da lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568250:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.568252:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756e00/0x4605c7c97e1c57da lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568257:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756e00/0x4605c7c97e1c57da lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568275:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756800/0x4605c7c97e1c57e1 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568280:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756800/0x4605c7c97e1c57e1 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568285:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756800/0x4605c7c97e1c57e1 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568290:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756800/0x4605c7c97e1c57e1 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568296:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756800/0x4605c7c97e1c57e1 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568301:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902358756800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.568304:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756800/0x4605c7c97e1c57e1 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568310:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756800/0x4605c7c97e1c57e1 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568315:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756800/0x4605c7c97e1c57e1 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568320:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.568323:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756800/0x4605c7c97e1c57e1 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568329:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756800/0x4605c7c97e1c57e1 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568339:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757e00/0x4605c7c97e1c57e8 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568702:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757e00/0x4605c7c97e1c57e8 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568708:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757e00/0x4605c7c97e1c57e8 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568715:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757e00/0x4605c7c97e1c57e8 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568721:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757e00/0x4605c7c97e1c57e8 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568726:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902358757e00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.568729:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757e00/0x4605c7c97e1c57e8 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568734:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757e00/0x4605c7c97e1c57e8 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568740:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757e00/0x4605c7c97e1c57e8 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568746:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.568749:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757e00/0x4605c7c97e1c57e8 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568755:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757e00/0x4605c7c97e1c57e8 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568769:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756400/0x4605c7c97e1c57ef lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568775:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756400/0x4605c7c97e1c57ef lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568781:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756400/0x4605c7c97e1c57ef lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568787:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756400/0x4605c7c97e1c57ef lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568792:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756400/0x4605c7c97e1c57ef lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568797:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902358756400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.568800:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756400/0x4605c7c97e1c57ef lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568805:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756400/0x4605c7c97e1c57ef lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568810:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756400/0x4605c7c97e1c57ef lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568815:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.568817:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756400/0x4605c7c97e1c57ef lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568823:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756400/0x4605c7c97e1c57ef lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568832:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757400/0x4605c7c97e1c57f6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568838:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757400/0x4605c7c97e1c57f6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568844:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757400/0x4605c7c97e1c57f6 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568850:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757400/0x4605c7c97e1c57f6 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568855:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757400/0x4605c7c97e1c57f6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568860:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902358757400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.568862:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757400/0x4605c7c97e1c57f6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568867:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757400/0x4605c7c97e1c57f6 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568873:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757400/0x4605c7c97e1c57f6 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568904:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.568907:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757400/0x4605c7c97e1c57f6 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568911:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757400/0x4605c7c97e1c57f6 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568921:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756600/0x4605c7c97e1c57fd lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568926:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756600/0x4605c7c97e1c57fd lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568931:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756600/0x4605c7c97e1c57fd lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568936:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756600/0x4605c7c97e1c57fd lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568942:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756600/0x4605c7c97e1c57fd lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568946:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902358756600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.568949:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756600/0x4605c7c97e1c57fd lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568955:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756600/0x4605c7c97e1c57fd lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568961:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756600/0x4605c7c97e1c57fd lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568966:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.568969:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756600/0x4605c7c97e1c57fd lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568974:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358756600/0x4605c7c97e1c57fd lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568987:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757000/0x4605c7c97e1c5804 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568993:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757000/0x4605c7c97e1c5804 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.568999:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757000/0x4605c7c97e1c5804 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569004:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757000/0x4605c7c97e1c5804 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569010:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757000/0x4605c7c97e1c5804 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569015:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902358757000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.569017:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757000/0x4605c7c97e1c5804 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569023:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757000/0x4605c7c97e1c5804 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569029:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757000/0x4605c7c97e1c5804 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569034:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.569038:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757000/0x4605c7c97e1c5804 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569044:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902358757000/0x4605c7c97e1c5804 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569054:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d800/0x4605c7c97e1c580b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569119:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d800/0x4605c7c97e1c580b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569126:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d800/0x4605c7c97e1c580b lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569132:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d800/0x4605c7c97e1c580b lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569138:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d800/0x4605c7c97e1c580b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569143:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265d800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.569146:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d800/0x4605c7c97e1c580b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569151:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d800/0x4605c7c97e1c580b lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569260:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d800/0x4605c7c97e1c580b lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569266:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.569279:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d800/0x4605c7c97e1c580b lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569284:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d800/0x4605c7c97e1c580b lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569299:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5812 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569304:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5812 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569310:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5812 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569315:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5812 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569321:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5812 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569325:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265c800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.569328:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5812 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569333:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5812 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569338:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5812 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569406:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.569411:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5812 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.569416:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5812 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201389.569443:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442746368:192.168.201.130@tcp:63:ll_sa.0 00000100:00100000:1.0:1788201389.577160:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442746752:192.168.201.130@tcp:36:rm.0 00000080:00200000:1.0:1788201389.577264:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201389.577287:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201389.577347:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201389.577362:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x41e:0x0](ffff9023449a0908) 00000008:00010000:1.0:1788201389.577399:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201389.577487:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.577506:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.577519:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00010000:00010000:1.0:1788201389.577532:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00000080:00200000:1.0:1788201389.577541:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.577547:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.577551:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.577558:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:1.0:1788201389.577562:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00010000:00010000:1.0:1788201389.577570:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00000080:00200000:1.0:1788201389.577585:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.577592:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:1.0:1788201389.577597:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.577601:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-2, flags=8 00000080:00200008:1.0:1788201389.577743:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-2:[0x200000bd1:0x40d:0x0](ffff9023449a6c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201389.577749:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x40d:0x0](ffff9023449a6c08),name=t-2 00000002:00010000:1.0:1788201389.577762:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x40d:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.577770:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.577779:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.577786:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.577790:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x40d:0x0](ffff9023449a6c08) 00000080:00010000:1.0:1788201389.577793:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x40d:0x0](ffff9023449a6c08) for lock 0x4605c7c97e1c5381 00000080:00010000:1.0:1788201389.577797:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5381 from it ffffa0b181c3bcb8 00010000:00010000:1.0:1788201389.577799:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.577808:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201389.577832:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x40d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788201389.577839:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x40d:0x0] 00000008:00010000:1.0:1788201389.577857:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788201389.577862:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023586d2630, osc lock ffff90235090cd10, flags 0x400001002 00010000:00010000:1.0:1788201389.577931:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.577942:0:17859:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788201389.578341:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1788201389.578355:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201389.578375:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442746880:192.168.201.130@tcp:101:rm.0 00010000:00010000:0.0:1788201389.578909:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c5827 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.578925:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c5827 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201389.579200:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201389.579213:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442747008:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:1.0:1788201389.582337:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.582362:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d59af expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.582377:0:2360:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d59af expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.582389:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d59af expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201389.582405:0:2360:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d59af expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788201389.582418:0:2360:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1788201389.582439:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d59af expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.582449:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d59af expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.582463:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d59af expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.582472:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d59af expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201389.582492:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442746880:192.168.201.130@tcp:101:rm.0 00000080:00200000:2.0:1788201389.582584:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x40d:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201389.582590:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235844f530 (ffff9023500e7a28/0) at cl_lock_release():245 00000020:00010000:2.0:1788201389.582593:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235844f530 (ffff9023500e7a28/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.582598:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023586d2630 (ffff9023500e7e18/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.582601:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235844f530 (ffff9023500e7a28/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201389.582613:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023586d2630 (ffff9023500e7a28/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201389.582651:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x40d:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201389.582708:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201389.582726:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.582740:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00010000:00010000:2.0:1788201389.582764:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00000080:00200000:2.0:1788201389.582773:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.582781:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201389.582785:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201389.582791:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:2.0:1788201389.582794:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00010000:00010000:2.0:1788201389.582802:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 57 lru_type: 1 00000080:00200000:2.0:1788201389.582813:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.582820:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:2.0:1788201389.582825:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.582831:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-2, flags=0 00000080:00200000:2.0:1788201389.582835:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:2.0:1788201389.582840:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-2, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:2.0:1788201389.582929:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201389.582952:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x40d:0x0]: losing layout lock 00010000:00010000:2.0:1788201389.583113:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201389.583122:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201389.583126:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730000/0x4605c7c97e1c5381 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x40d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5396 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201389.583264:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201389.583284:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234518d880 x1875064442747264/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201389.583306:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518d880 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442747264:192.168.201.130@tcp:36:rm.0 00000100:00100000:2.0:1788201389.583400:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3840 going to sleep for 11 seconds 00010000:00010000:0.0:1788201389.589860:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c5827 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.589920:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c5827 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.589928:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c5827 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.589935:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c5827 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.589944:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c5827 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.589950:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865b400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201389.589955:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c5827 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.589963:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c5827 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.589970:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c5827 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.589981:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201389.589987:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c5827 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.589993:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c5827 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201389.590022:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442747008:192.168.201.130@tcp:63:ll_sa.0 00000100:00100000:2.0:1788201389.591504:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518d880 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442747264:192.168.201.130@tcp:36:rm.0 00000080:00200000:2.0:1788201389.591586:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x40d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201389.591597:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201389.591663:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x40d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201389.591683:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x40d:0x0](ffff9023449a6c08) 00000008:00010000:2.0:1788201389.591719:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x463:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d59af expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201389.592180:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201389.592197:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.592210:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 57 lru_type: 1 00010000:00010000:2.0:1788201389.592221:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 57 lru_type: 1 00000080:00200000:2.0:1788201389.592230:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.592235:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201389.592238:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201389.592245:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:2.0:1788201389.592248:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 57 lru_type: 1 00010000:00010000:2.0:1788201389.592255:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 58 lru_type: 1 00000080:00200000:2.0:1788201389.592280:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.592286:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:2.0:1788201389.592291:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.592295:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-20, flags=8 00010000:00010000:2.0:1788201389.592305:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1788201389.592312:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53ff from it ffffa0b181c3baf8 00010000:00010000:2.0:1788201389.592315:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201389.592322:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1788201389.592339:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-20:[0x200000bd1:0x41f:0x0](ffff9023449a5348), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201389.592344:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x41f:0x0](ffff9023449a5348),name=t-20 00000002:00010000:2.0:1788201389.592355:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x41f:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.592362:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201389.592370:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201389.592377:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.592381:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x41f:0x0](ffff9023449a5348) 00000080:00010000:2.0:1788201389.592383:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x41f:0x0](ffff9023449a5348) for lock 0x4605c7c97e1c53ff 00000080:00010000:2.0:1788201389.592387:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53ff from it ffffa0b181c3bcb8 00010000:00010000:2.0:1788201389.592390:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201389.592397:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201389.592420:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41f:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201389.592426:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x41f:0x0] 00000008:00010000:2.0:1788201389.592443:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201389.592446:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398a30, osc lock ffff9023500dd8f0, flags 0x400001002 00010000:00010000:2.0:1788201389.592460:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201389.592470:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788201389.592483:0:17859:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788201389.592493:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788201389.592497:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201389.592505:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1788201389.592514:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788201389.592524:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x41f:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201389.592527:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201389.592530:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.592533:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398a30 (ffff902350b9df38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.592537:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201389.592539:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398a30 (ffff902350b9d1b8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201389.592555:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41f:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201389.592578:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201389.592584:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.592591:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 58 lru_type: 1 00010000:00010000:2.0:1788201389.592600:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 58 lru_type: 1 00000080:00200000:2.0:1788201389.596355:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.596369:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201389.596375:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201389.596385:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:2.0:1788201389.596392:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 58 lru_type: 1 00010000:00010000:2.0:1788201389.596402:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 59 lru_type: 1 00000080:00200000:2.0:1788201389.596420:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.596431:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:2.0:1788201389.596440:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.596451:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-20, flags=0 00000080:00200000:2.0:1788201389.596455:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:2.0:1788201389.596459:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-20, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:2.0:1788201389.596548:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201389.596575:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x41f:0x0]: losing layout lock 00010000:00010000:2.0:1788201389.596632:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201389.596639:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201389.596642:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802600/0x4605c7c97e1c53ff lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x41f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d560c expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:2.0:1788201389.596670:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201389.596683:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234518d180 x1875064442747520/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201389.596709:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518d180 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442747520:192.168.201.130@tcp:36:rm.0 00000100:00100000:2.0:1788201389.596810:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3840 going to sleep for 11 seconds 00000100:00100000:2.0:1788201389.604279:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518d180 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442747520:192.168.201.130@tcp:36:rm.0 00000080:00200000:2.0:1788201389.604508:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x41f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201389.604520:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201389.604574:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x41f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201389.604588:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x41f:0x0](ffff9023449a5348) 00000008:00010000:2.0:1788201389.604623:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201389.604703:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201389.604866:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.605325:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 59 lru_type: 1 00010000:00010000:2.0:1788201389.605341:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 59 lru_type: 1 00000080:00200000:2.0:1788201389.605350:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.605357:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201389.605360:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201389.605368:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:2.0:1788201389.605470:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 59 lru_type: 1 00010000:00010000:2.0:1788201389.605480:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 60 lru_type: 1 00000080:00200000:2.0:1788201389.605501:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.605508:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:2.0:1788201389.605514:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.605517:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-21, flags=8 00010000:00010000:2.0:1788201389.605528:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1788201389.605536:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5406 from it ffffa0b181c3baf8 00010000:00010000:2.0:1788201389.605539:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201389.605545:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1788201389.605561:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-21:[0x200000bd1:0x420:0x0](ffff9023449a1148), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201389.605566:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x420:0x0](ffff9023449a1148),name=t-21 00000002:00010000:2.0:1788201389.605577:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x420:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.605584:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201389.605592:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201389.605598:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.605602:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x420:0x0](ffff9023449a1148) 00000080:00010000:2.0:1788201389.605605:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x420:0x0](ffff9023449a1148) for lock 0x4605c7c97e1c5406 00000080:00010000:2.0:1788201389.605608:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5406 from it ffffa0b181c3bcb8 00010000:00010000:2.0:1788201389.605611:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201389.605618:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201389.605645:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x420:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201389.605651:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x420:0x0] 00000008:00010000:2.0:1788201389.605673:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201389.605676:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398a30, osc lock ffff9023500dd8f0, flags 0x400001002 00010000:00010000:2.0:1788201389.605691:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201389.605701:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1788201389.605775:0:17859:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:2.0:1788201389.605978:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788201389.605983:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201389.605990:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1788201389.606005:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788201389.606015:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x420:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201389.606019:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201389.606021:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.606024:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398a30 (ffff902350b9df38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.606027:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc62b0 (ffff902350b9d1b8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201389.606030:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398a30 (ffff902350b9d1b8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201389.606052:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x420:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201389.606081:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201389.606089:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.606097:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 60 lru_type: 1 00010000:00010000:2.0:1788201389.606105:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 60 lru_type: 1 00000080:00200000:2.0:1788201389.606112:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.606116:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201389.606119:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201389.606122:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:2.0:1788201389.606125:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 60 lru_type: 1 00010000:00010000:2.0:1788201389.606132:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00000080:00200000:2.0:1788201389.606146:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.606217:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:2.0:1788201389.606223:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.606230:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-21, flags=0 00000080:00200000:2.0:1788201389.606233:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:2.0:1788201389.606238:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-21, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:2.0:1788201389.606350:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201389.606491:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x420:0x0]: losing layout lock 00010000:00010000:2.0:1788201389.606541:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201389.606549:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201389.606551:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802a00/0x4605c7c97e1c5406 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x420:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d562f expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:2.0:1788201389.606577:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201389.606590:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234518df80 x1875064442747776/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201389.606613:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518df80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442747776:192.168.201.130@tcp:36:rm.0 00000100:00100000:2.0:1788201389.608211:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3840 going to sleep for 11 seconds 00000100:00100000:2.0:1788201389.615492:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518df80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442747776:192.168.201.130@tcp:36:rm.0 00000080:00200000:2.0:1788201389.615584:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x420:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201389.615598:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201389.615841:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x420:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.615862:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x420:0x0](ffff9023449a1148) 00000008:00010000:0.0:1788201389.615929:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.616013:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.616031:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.616045:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00010000:00010000:0.0:1788201389.616057:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00000080:00200000:0.0:1788201389.616064:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.616070:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.616073:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.616080:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:0.0:1788201389.616084:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00010000:00010000:0.0:1788201389.616094:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 62 lru_type: 1 00000080:00200000:0.0:1788201389.616111:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.616116:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.616121:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.616125:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-22, flags=8 00010000:00010000:0.0:1788201389.616134:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1788201389.616141:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c540d from it ffffa0b181c3baf8 00010000:00010000:0.0:1788201389.616144:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.616151:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:0.0:1788201389.616166:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-22:[0x200000bd1:0x421:0x0](ffff9023449a5b88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201389.616171:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x421:0x0](ffff9023449a5b88),name=t-22 00000002:00010000:0.0:1788201389.616182:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x421:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.616199:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.616207:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201389.616213:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.616217:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x421:0x0](ffff9023449a5b88) 00000080:00010000:0.0:1788201389.616219:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x421:0x0](ffff9023449a5b88) for lock 0x4605c7c97e1c540d 00000080:00010000:0.0:1788201389.616223:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c540d from it ffffa0b181c3bcb8 00010000:00010000:0.0:1788201389.616226:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.616232:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.616262:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x421:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.616280:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x421:0x0] 00000008:00010000:0.0:1788201389.616303:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201389.616307:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902350050930, osc lock ffff902343fb7db0, flags 0x400001002 00010000:00010000:0.0:1788201389.616323:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.616333:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1788201389.616346:0:17859:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1788201389.616356:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788201389.616360:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.616367:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1788201389.616378:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788201389.616396:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x421:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201389.616401:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.616404:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.616407:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350050930 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.616411:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.616414:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350050930 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.616438:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x421:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.616460:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.616466:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.616473:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 62 lru_type: 1 00010000:00010000:0.0:1788201389.616481:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 62 lru_type: 1 00000080:00200000:0.0:1788201389.616487:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.616491:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.616493:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.616497:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:0.0:1788201389.616500:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 62 lru_type: 1 00010000:00010000:0.0:1788201389.616506:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 63 lru_type: 1 00000080:00200000:0.0:1788201389.616520:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.616525:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.616528:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.616534:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-22, flags=0 00000080:00200000:0.0:1788201389.616536:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.616541:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-22, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:0.0:1788201389.616578:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.616601:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x421:0x0]: losing layout lock 00010000:00010000:0.0:1788201389.616644:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201389.616650:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201389.616653:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345802e00/0x4605c7c97e1c540d lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x421:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5652 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:0.0:1788201389.616677:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201389.616688:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450ba680 x1875064442748032/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201389.616710:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442748032:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201389.616785:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436c09c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.621058:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442748032:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201389.621155:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x421:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201389.621168:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201389.621233:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x421:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.621244:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x421:0x0](ffff9023449a5b88) 00000008:00010000:0.0:1788201389.621288:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.621381:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.621412:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.621430:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 63 lru_type: 1 00010000:00010000:0.0:1788201389.621442:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 63 lru_type: 1 00000080:00200000:0.0:1788201389.621451:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.621456:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.621460:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.621467:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:0.0:1788201389.621471:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 63 lru_type: 1 00010000:00010000:0.0:1788201389.621481:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 64 lru_type: 1 00000080:00200000:0.0:1788201389.621498:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.621506:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.621511:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.621513:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-23, flags=8 00010000:00010000:0.0:1788201389.621522:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1788201389.621532:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5414 from it ffffa0b181c3baf8 00010000:00010000:0.0:1788201389.621534:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.621538:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:0.0:1788201389.621552:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-23:[0x200000bd1:0x422:0x0](ffff9023449b42c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201389.621555:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x422:0x0](ffff9023449b42c8),name=t-23 00000002:00010000:0.0:1788201389.621564:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x422:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.621569:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.621574:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201389.621578:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.621580:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x422:0x0](ffff9023449b42c8) 00000080:00010000:0.0:1788201389.621582:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x422:0x0](ffff9023449b42c8) for lock 0x4605c7c97e1c5414 00000080:00010000:0.0:1788201389.621584:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5414 from it ffffa0b181c3bcb8 00010000:00010000:0.0:1788201389.621586:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.621590:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.621608:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x422:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.621613:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x422:0x0] 00000008:00010000:0.0:1788201389.621626:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201389.621628:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902350050930, osc lock ffff902343fb7db0, flags 0x400001002 00010000:00010000:0.0:1788201389.621642:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.621648:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1788201389.621656:0:17859:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1788201389.621663:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788201389.621665:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.621670:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1788201389.621675:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788201389.621684:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x422:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201389.621686:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.621690:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.621693:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350050930 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.621697:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.621699:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350050930 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.621716:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x422:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.621738:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.621744:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.621751:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 64 lru_type: 1 00010000:00010000:0.0:1788201389.621759:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 64 lru_type: 1 00000080:00200000:0.0:1788201389.621765:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.621769:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.621772:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.621775:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:0.0:1788201389.621778:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 64 lru_type: 1 00010000:00010000:0.0:1788201389.621785:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 65 lru_type: 1 00000080:00200000:0.0:1788201389.621792:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.621796:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.621800:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.621805:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-23, flags=0 00000080:00200000:0.0:1788201389.621808:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.621814:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-23, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:0.0:1788201389.621851:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.621872:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x422:0x0]: losing layout lock 00010000:00010000:0.0:1788201389.621939:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201389.621946:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201389.621949:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730200/0x4605c7c97e1c5414 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x422:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5675 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:0.0:1788201389.621973:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201389.621983:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450b9880 x1875064442748288/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201389.622006:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442748288:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201389.622351:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436c09c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.631232:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442748288:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201389.631329:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x422:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201389.631344:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201389.631414:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x422:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.631428:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x422:0x0](ffff9023449b42c8) 00000008:00010000:0.0:1788201389.631465:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.631554:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.631571:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.631586:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 65 lru_type: 1 00010000:00010000:0.0:1788201389.631598:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 65 lru_type: 1 00000080:00200000:0.0:1788201389.631605:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.631611:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.631615:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.631622:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:0.0:1788201389.631626:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 65 lru_type: 1 00010000:00010000:0.0:1788201389.631634:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00000080:00200000:0.0:1788201389.631648:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.631654:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.631660:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.631663:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-24, flags=8 00010000:00010000:0.0:1788201389.631673:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1788201389.631679:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c541b from it ffffa0b181c3baf8 00010000:00010000:0.0:1788201389.631682:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.631688:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:0.0:1788201389.631708:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-24:[0x200000bd1:0x423:0x0](ffff9023449b2a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201389.631723:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x423:0x0](ffff9023449b2a08),name=t-24 00000002:00010000:0.0:1788201389.631735:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x423:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.631742:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.631751:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201389.631756:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.631761:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x423:0x0](ffff9023449b2a08) 00000080:00010000:0.0:1788201389.631763:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x423:0x0](ffff9023449b2a08) for lock 0x4605c7c97e1c541b 00000080:00010000:0.0:1788201389.631767:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c541b from it ffffa0b181c3bcb8 00010000:00010000:0.0:1788201389.631770:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.631778:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.631802:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x423:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.631808:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x423:0x0] 00000008:00010000:0.0:1788201389.631827:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201389.631831:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902350050930, osc lock ffff902343fb7db0, flags 0x400001002 00010000:00010000:0.0:1788201389.631847:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.631862:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1788201389.631901:0:17859:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1788201389.631913:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788201389.631917:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.631924:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1788201389.631935:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788201389.631945:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x423:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201389.631949:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.631952:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.631955:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350050930 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.631959:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.631961:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350050930 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.631979:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x423:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.632001:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.632007:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.632014:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00010000:00010000:0.0:1788201389.632023:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00000080:00200000:0.0:1788201389.632029:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.632033:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.632035:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.632039:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:0.0:1788201389.632042:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00010000:00010000:0.0:1788201389.632049:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 67 lru_type: 1 00000080:00200000:0.0:1788201389.632057:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.632061:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.632065:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.632070:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-24, flags=0 00000080:00200000:0.0:1788201389.632073:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.632077:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-24, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:0.0:1788201389.632113:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.632135:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x423:0x0]: losing layout lock 00010000:00010000:0.0:1788201389.632174:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201389.632182:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201389.632185:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730e00/0x4605c7c97e1c541b lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x423:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5698 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:0.0:1788201389.632209:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201389.632219:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450b8700 x1875064442748544/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201389.632241:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442748544:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201389.638050:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436c09c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.642837:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442748544:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201389.642956:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x423:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201389.642968:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201389.643026:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x423:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.643039:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x423:0x0](ffff9023449b2a08) 00000008:00010000:0.0:1788201389.643075:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.643173:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.643198:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.643211:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 67 lru_type: 1 00010000:00010000:0.0:1788201389.643224:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 67 lru_type: 1 00000080:00200000:0.0:1788201389.643230:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.643236:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.643239:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.643247:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:0.0:1788201389.643251:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 67 lru_type: 1 00010000:00010000:0.0:1788201389.643259:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 68 lru_type: 1 00000080:00200000:0.0:1788201389.643307:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.643314:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.643319:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.643322:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-25, flags=8 00010000:00010000:0.0:1788201389.643333:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1788201389.643341:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5422 from it ffffa0b181c3baf8 00010000:00010000:0.0:1788201389.643344:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.643350:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:0.0:1788201389.643368:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-25:[0x200000bd1:0x424:0x0](ffff9023449b6c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201389.643378:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x424:0x0](ffff9023449b6c08),name=t-25 00000002:00010000:0.0:1788201389.643398:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x424:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.643405:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.643417:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201389.643423:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.643426:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x424:0x0](ffff9023449b6c08) 00000080:00010000:0.0:1788201389.643429:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x424:0x0](ffff9023449b6c08) for lock 0x4605c7c97e1c5422 00000080:00010000:0.0:1788201389.643432:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5422 from it ffffa0b181c3bcb8 00010000:00010000:0.0:1788201389.643435:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.643441:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.643465:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x424:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.643471:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x424:0x0] 00000008:00010000:0.0:1788201389.643491:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201389.643494:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902350050930, osc lock ffff902343fb7db0, flags 0x400001002 00010000:00010000:0.0:1788201389.643509:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.643520:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1788201389.643534:0:17859:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1788201389.643550:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788201389.643554:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.643561:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1788201389.643571:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788201389.643580:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x424:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201389.643583:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.643586:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.643588:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350050930 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.643592:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.643595:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350050930 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.643612:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x424:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.643634:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.643641:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.643648:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 68 lru_type: 1 00010000:00010000:0.0:1788201389.643656:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 68 lru_type: 1 00000080:00200000:0.0:1788201389.643663:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.643667:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.643670:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.643674:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:0.0:1788201389.643677:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 68 lru_type: 1 00010000:00010000:0.0:1788201389.643683:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 69 lru_type: 1 00000080:00200000:0.0:1788201389.643691:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.643695:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.643698:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.643703:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-25, flags=0 00000080:00200000:0.0:1788201389.643707:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.643711:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-25, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:0.0:1788201389.643747:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.643769:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x424:0x0]: losing layout lock 00010000:00010000:0.0:1788201389.643812:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201389.643819:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201389.643822:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730c00/0x4605c7c97e1c5422 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x424:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d56bb expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:0.0:1788201389.643845:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201389.643856:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450bb100 x1875064442748800/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201389.643911:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442748800:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201389.643984:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436c09c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.649528:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442748800:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201389.649613:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x424:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201389.649625:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201389.649684:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x424:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.649697:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x424:0x0](ffff9023449b6c08) 00000008:00010000:0.0:1788201389.649748:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.649840:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.649859:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.649873:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 69 lru_type: 1 00010000:00010000:0.0:1788201389.649912:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 69 lru_type: 1 00000080:00200000:0.0:1788201389.649921:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.649927:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.649931:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.649941:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:0.0:1788201389.649945:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 69 lru_type: 1 00010000:00010000:0.0:1788201389.649952:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00000080:00200000:0.0:1788201389.649973:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.649980:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.649985:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.649990:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-26, flags=8 00010000:00010000:0.0:1788201389.650002:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1788201389.650010:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5429 from it ffffa0b181c3baf8 00010000:00010000:0.0:1788201389.650013:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.650021:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:0.0:1788201389.650039:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-26:[0x200000bd1:0x425:0x0](ffff9023449a2a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201389.650044:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x425:0x0](ffff9023449a2a08),name=t-26 00000002:00010000:0.0:1788201389.650055:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x425:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.650062:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.650071:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201389.650078:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.650082:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x425:0x0](ffff9023449a2a08) 00000080:00010000:0.0:1788201389.650085:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x425:0x0](ffff9023449a2a08) for lock 0x4605c7c97e1c5429 00000080:00010000:0.0:1788201389.650089:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5429 from it ffffa0b181c3bcb8 00010000:00010000:0.0:1788201389.650092:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.650099:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.650125:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x425:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.650132:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x425:0x0] 00000008:00010000:0.0:1788201389.650151:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201389.650154:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902350050930, osc lock ffff902343fb7db0, flags 0x400001002 00010000:00010000:0.0:1788201389.650170:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.650180:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1788201389.650192:0:17859:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1788201389.650202:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788201389.650205:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.650213:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1788201389.650228:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788201389.650239:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x425:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201389.650243:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.650246:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.650248:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350050930 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.650252:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.650255:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350050930 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.650288:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x425:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.650327:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.650334:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.650341:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00010000:00010000:0.0:1788201389.650348:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00000080:00200000:0.0:1788201389.650354:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.650358:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.650360:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.650364:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:0.0:1788201389.650367:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00010000:00010000:0.0:1788201389.650374:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00000080:00200000:0.0:1788201389.650398:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.650403:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.650407:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.650413:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-26, flags=0 00000080:00200000:0.0:1788201389.650415:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.650419:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-26, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:0.0:1788201389.650450:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.650465:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x425:0x0]: losing layout lock 00010000:00010000:0.0:1788201389.650500:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201389.650505:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201389.650507:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343730800/0x4605c7c97e1c5429 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x425:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d56de expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:0.0:1788201389.650524:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201389.650531:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450bb480 x1875064442749056/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201389.650547:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442749056:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201389.650594:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436c09c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.656205:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442749056:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201389.656288:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x425:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201389.656297:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201389.656346:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x425:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.656356:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x425:0x0](ffff9023449a2a08) 00000008:00010000:0.0:1788201389.656395:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.656461:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.656476:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.656486:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00010000:00010000:0.0:1788201389.656495:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00000080:00200000:0.0:1788201389.656500:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.656504:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.656507:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.656512:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:0.0:1788201389.656515:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00010000:00010000:0.0:1788201389.656521:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00000080:00200000:0.0:1788201389.656532:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.656536:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.656540:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.656542:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-27, flags=8 00010000:00010000:0.0:1788201389.656549:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1788201389.656558:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5430 from it ffffa0b181c3baf8 00010000:00010000:0.0:1788201389.656560:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.656564:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:0.0:1788201389.656574:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-27:[0x200000bd1:0x426:0x0](ffff9023448e7448), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201389.656577:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x426:0x0](ffff9023448e7448),name=t-27 00000002:00010000:0.0:1788201389.656584:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x426:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.656588:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.656592:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201389.656596:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.656598:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x426:0x0](ffff9023448e7448) 00000080:00010000:0.0:1788201389.656599:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x426:0x0](ffff9023448e7448) for lock 0x4605c7c97e1c5430 00000080:00010000:0.0:1788201389.656602:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5430 from it ffffa0b181c3bcb8 00010000:00010000:0.0:1788201389.656603:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.656607:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.656624:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x426:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.656629:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x426:0x0] 00000008:00010000:0.0:1788201389.656641:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201389.656643:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902350050930, osc lock ffff902343fb7db0, flags 0x400001002 00010000:00010000:0.0:1788201389.656656:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.656663:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1788201389.656671:0:17859:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1788201389.656679:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788201389.656681:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.656685:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1788201389.656690:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788201389.656697:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x426:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201389.656699:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.656702:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.656704:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350050930 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.656707:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.656708:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350050930 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.656723:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x426:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.656743:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.656750:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.656756:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00010000:00010000:0.0:1788201389.656764:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00000080:00200000:0.0:1788201389.656771:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.656775:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.656777:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.656780:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:0.0:1788201389.656783:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00010000:00010000:0.0:1788201389.656789:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00000080:00200000:0.0:1788201389.656798:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.656802:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.656805:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.656810:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-27, flags=0 00000080:00200000:0.0:1788201389.656813:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.656818:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-27, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:0.0:1788201389.656855:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.656901:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x426:0x0]: losing layout lock 00010000:00010000:0.0:1788201389.656955:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201389.656963:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201389.656966:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803400/0x4605c7c97e1c5430 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x426:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5701 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:0.0:1788201389.656988:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201389.656999:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450bb800 x1875064442749312/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201389.657018:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442749312:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201389.657085:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436c09c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.662646:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442749312:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201389.662852:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x426:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201389.662863:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201389.663052:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x426:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.663066:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x426:0x0](ffff9023448e7448) 00000008:00010000:0.0:1788201389.663101:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.663171:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.663196:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.663207:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00010000:00010000:0.0:1788201389.663216:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00000080:00200000:0.0:1788201389.663221:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.663226:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.663228:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.663234:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:0.0:1788201389.663237:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00010000:00010000:0.0:1788201389.663243:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00000080:00200000:0.0:1788201389.663258:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.663264:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.663278:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.663282:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-28, flags=8 00010000:00010000:0.0:1788201389.663294:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1788201389.663302:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5437 from it ffffa0b181c3baf8 00010000:00010000:0.0:1788201389.663306:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.663313:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:0.0:1788201389.663334:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-28:[0x200000bd1:0x427:0x0](ffff9023448e1988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201389.663338:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x427:0x0](ffff9023448e1988),name=t-28 00000002:00010000:0.0:1788201389.663348:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x427:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.663354:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.663363:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201389.663379:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.663383:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x427:0x0](ffff9023448e1988) 00000080:00010000:0.0:1788201389.664142:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x427:0x0](ffff9023448e1988) for lock 0x4605c7c97e1c5437 00000080:00010000:0.0:1788201389.664149:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5437 from it ffffa0b181c3bcb8 00010000:00010000:0.0:1788201389.664152:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.664165:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.664200:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x427:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.664205:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x427:0x0] 00000008:00010000:0.0:1788201389.664221:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201389.664224:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902350050930, osc lock ffff902343fb7db0, flags 0x400001002 00010000:00010000:0.0:1788201389.664239:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x470:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.664253:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1788201389.664275:0:17859:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 4/1,0 mode: PR/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1788201389.664286:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788201389.664290:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.664298:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1788201389.664308:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788201389.664317:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x427:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201389.664322:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.664324:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.664326:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350050930 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.664329:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.664331:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350050930 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.664349:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x427:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.664397:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.664405:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.664413:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00010000:00010000:0.0:1788201389.664421:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00000080:00200000:0.0:1788201389.664428:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.664432:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.664435:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.664439:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:0.0:1788201389.664441:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00010000:00010000:0.0:1788201389.664446:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00000080:00200000:0.0:1788201389.664451:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.664454:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.664457:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.664461:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-28, flags=0 00000080:00200000:0.0:1788201389.664463:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.664467:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-28, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:0.0:1788201389.664495:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.664515:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x427:0x0]: losing layout lock 00010000:00010000:0.0:1788201389.664557:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201389.664564:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201389.664567:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803e00/0x4605c7c97e1c5437 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x427:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5724 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:0.0:1788201389.664591:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201389.664603:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450bbb80 x1875064442749568/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201389.664625:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442749568:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201389.664700:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436c09c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.673411:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442749568:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201389.673507:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x427:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201389.673518:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201389.673573:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x427:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.673586:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x427:0x0](ffff9023448e1988) 00000008:00010000:0.0:1788201389.673625:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x470:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.673710:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.673727:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.673742:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00010000:00010000:0.0:1788201389.673755:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00000080:00200000:0.0:1788201389.673762:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.673769:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.673773:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.673780:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:0.0:1788201389.673784:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00010000:00010000:0.0:1788201389.673792:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 76 lru_type: 1 00000080:00200000:0.0:1788201389.673809:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.673817:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.673821:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.673825:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-29, flags=8 00010000:00010000:0.0:1788201389.673839:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1788201389.673847:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c543e from it ffffa0b181c3baf8 00010000:00010000:0.0:1788201389.673851:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201389.673859:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200008:0.0:1788201389.673916:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-29:[0x200000bd1:0x428:0x0](ffff9023448e4b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201389.673921:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x428:0x0](ffff9023448e4b08),name=t-29 00000002:00010000:0.0:1788201389.673932:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x428:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.673940:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.673953:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201389.673961:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.673965:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x428:0x0](ffff9023448e4b08) 00000080:00010000:0.0:1788201389.673968:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x428:0x0](ffff9023448e4b08) for lock 0x4605c7c97e1c543e 00000080:00010000:0.0:1788201389.673972:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c543e from it ffffa0b181c3bcb8 00010000:00010000:0.0:1788201389.673976:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201389.673983:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.674008:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x428:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.674014:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x428:0x0] 00000008:00010000:0.0:1788201389.674035:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201389.674039:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902350050930, osc lock ffff902343fb7db0, flags 0x400001002 00010000:00010000:0.0:1788201389.674057:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x470:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.674068:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:0.0:1788201389.674083:0:17859:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 4/1,0 mode: PR/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:0.0:1788201389.674098:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788201389.674101:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.674110:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:0.0:1788201389.674121:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1788201389.674132:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x428:0x0] updating i_size 0 i_blocks 1 00000020:00010000:0.0:1788201389.674136:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:0.0:1788201389.674139:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.674142:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350050930 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:0.0:1788201389.674146:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:0.0:1788201389.674551:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350050930 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:0.0:1788201389.674575:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x428:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.674606:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.674615:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.674623:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 76 lru_type: 1 00010000:00010000:0.0:1788201389.674633:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 76 lru_type: 1 00000080:00200000:0.0:1788201389.674639:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.674644:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.674648:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.674653:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:0.0:1788201389.674656:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 76 lru_type: 1 00010000:00010000:0.0:1788201389.674664:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 77 lru_type: 1 00000080:00200000:0.0:1788201389.674674:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.674679:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.674682:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.674688:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-29, flags=0 00000080:00200000:0.0:1788201389.674692:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:0.0:1788201389.674696:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-29, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:0.0:1788201389.674737:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201389.674760:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x428:0x0]: losing layout lock 00010000:00010000:0.0:1788201389.674803:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788201389.674810:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201389.674814:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90234839d200/0x4605c7c97e1c543e lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x428:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5747 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000020:00100000:0.0:1788201389.674839:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201389.674850:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450b8e00 x1875064442749824/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201389.674873:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442749824:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201389.674981:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436c09c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201389.682222:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442749824:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201389.682320:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x428:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201389.682333:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201389.682407:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x428:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201389.682421:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x428:0x0](ffff9023448e4b08) 00000008:00010000:0.0:1788201389.682459:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x470:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.682540:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201389.682555:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.682568:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 77 lru_type: 1 00010000:00010000:0.0:1788201389.682582:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 77 lru_type: 1 00000080:00200000:0.0:1788201389.682589:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.682595:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201389.682598:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:0.0:1788201389.682604:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:0.0:1788201389.682608:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 77 lru_type: 1 00010000:00010000:0.0:1788201389.682615:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 78 lru_type: 1 00000080:00200000:0.0:1788201389.682633:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.682639:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:0.0:1788201389.682643:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:0.0:1788201389.682647:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-3, flags=8 00000080:00200008:0.0:1788201389.682676:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-3:[0x200000bd1:0x40e:0x0](ffff9023449a63c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201389.682682:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x40e:0x0](ffff9023449a63c8),name=t-3 00000002:00010000:0.0:1788201389.682695:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x40e:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201389.682704:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.682712:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201389.682717:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201389.682721:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x40e:0x0](ffff9023449a63c8) 00000080:00010000:0.0:1788201389.682723:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x40e:0x0](ffff9023449a63c8) for lock 0x4605c7c97e1c5388 00000080:00010000:0.0:1788201389.682726:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5388 from it ffffa0b181c3bcb8 00010000:00010000:0.0:1788201389.682728:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201389.682735:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201389.682759:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x40e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201389.682766:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x40e:0x0] 00000008:00010000:0.0:1788201389.682783:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201389.682787:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902350050930, osc lock ffff902343fb7db0, flags 0x400001002 00010000:00010000:0.0:1788201389.682834:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 3/1,0 mode: --/PR res: [0x280000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.682843:0:17859:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 3/1,0 mode: --/PR res: [0x280000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788201389.682915:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1788201389.682936:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442749952:192.168.201.130@tcp:101:rm.0 00010000:00010000:1.0:1788201389.683167:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c5835 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683175:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c5835 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683196:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265de00/0x4605c7c97e1c583c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683202:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265de00/0x4605c7c97e1c583c lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683217:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c600/0x4605c7c97e1c5843 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683223:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c600/0x4605c7c97e1c5843 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683239:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c584a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 6 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683250:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c584a lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683279:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5851 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 7 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683286:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5851 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683304:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c5858 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 8 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683310:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c5858 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683324:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c585f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 9 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683330:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c585f lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683348:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5866 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 10 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683354:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5866 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683370:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c586d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 11 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683377:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c586d lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683409:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5874 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 12 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.683415:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5874 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.683465:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442750080:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:0.0:1788201389.690823:0:2359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.690849:0:2359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5aff expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.690863:0:2359:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5aff expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.690931:0:2359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5aff expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788201389.690948:0:2359:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5aff expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788201389.690959:0:2359:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1788201389.690968:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5aff expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.690976:0:2359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5aff expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.690984:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5aff expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.690992:0:2359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5aff expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788201389.691009:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442749952:192.168.201.130@tcp:101:rm.0 00000080:00200000:1.0:1788201389.691096:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x40e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:1.0:1788201389.691102:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_release():245 00000020:00010000:1.0:1788201389.691105:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201389.691110:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350050930 (ffff902348a2fb48/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788201389.691114:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372060 (ffff902348a2f098/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201389.691117:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350050930 (ffff902348a2f098/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201389.691164:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x40e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201389.691213:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.691229:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.691245:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 78 lru_type: 1 00010000:00010000:1.0:1788201389.691256:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 78 lru_type: 1 00000080:00200000:1.0:1788201389.691263:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.691281:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.691285:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.691290:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:1.0:1788201389.691294:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 78 lru_type: 1 00010000:00010000:1.0:1788201389.691302:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00000080:00200000:1.0:1788201389.691313:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.691320:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:1.0:1788201389.691325:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.691332:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-3, flags=0 00000080:00200000:1.0:1788201389.691336:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201389.691340:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-3, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:1.0:1788201389.691417:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201389.691441:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x40e:0x0]: losing layout lock 00010000:00010000:1.0:1788201389.691483:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201389.691490:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201389.691493:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803800/0x4605c7c97e1c5388 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x40e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d53b9 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201389.691517:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201389.691526:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234519d500 x1875064442750336/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201389.691549:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d500 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442750336:192.168.201.130@tcp:36:rm.0 00000100:00100000:1.0:1788201389.691634:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235312ad80 going to sleep for 11 seconds 00010000:00010000:1.0:1788201389.693735:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c5835 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693752:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c5835 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693758:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c5835 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693764:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c5835 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693773:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c5835 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 12 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693779:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265d000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.693783:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c5835 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 12 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693790:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c5835 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 12 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693797:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c5835 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 12 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693803:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.693810:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c5835 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 12 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693816:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c5835 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 12 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693839:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265de00/0x4605c7c97e1c583c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693845:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265de00/0x4605c7c97e1c583c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693850:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265de00/0x4605c7c97e1c583c lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693855:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265de00/0x4605c7c97e1c583c lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693860:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265de00/0x4605c7c97e1c583c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693864:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265de00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.693866:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265de00/0x4605c7c97e1c583c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693871:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265de00/0x4605c7c97e1c583c lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693913:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265de00/0x4605c7c97e1c583c lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693918:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.693922:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265de00/0x4605c7c97e1c583c lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693928:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265de00/0x4605c7c97e1c583c lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693942:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c600/0x4605c7c97e1c5843 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693949:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c600/0x4605c7c97e1c5843 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693953:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c600/0x4605c7c97e1c5843 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693958:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c600/0x4605c7c97e1c5843 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693963:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c600/0x4605c7c97e1c5843 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693968:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265c600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.693970:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c600/0x4605c7c97e1c5843 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693975:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c600/0x4605c7c97e1c5843 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693980:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c600/0x4605c7c97e1c5843 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693984:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.693987:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c600/0x4605c7c97e1c5843 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.693992:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c600/0x4605c7c97e1c5843 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694003:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c584a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694010:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c584a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694016:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c584a lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694022:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c584a lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694027:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c584a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694032:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265c400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.694034:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c584a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694039:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c584a lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694044:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c584a lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694048:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.694051:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c584a lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694056:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c584a lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694066:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5851 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694072:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5851 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694077:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5851 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694083:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5851 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694088:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5851 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694092:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265d600 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.694095:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5851 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694100:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5851 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694105:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5851 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694109:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.694112:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5851 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694116:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5851 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694126:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c5858 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694131:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c5858 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694135:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c5858 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694140:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c5858 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694145:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c5858 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694148:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265c200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.694151:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c5858 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694155:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c5858 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694160:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c5858 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694165:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.694167:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c5858 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694172:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c5858 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694182:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c585f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694187:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c585f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694192:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c585f lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694198:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c585f lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694203:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c585f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694208:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265d400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.694211:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c585f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694216:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c585f lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694223:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c585f lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694228:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.694231:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c585f lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694236:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c585f lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694246:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5866 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694253:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5866 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694258:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5866 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694263:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5866 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694279:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5866 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694285:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265ce00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.694288:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5866 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694293:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5866 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694299:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5866 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694304:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.694307:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5866 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694312:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5866 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694322:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c586d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694328:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c586d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694332:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c586d lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694338:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c586d lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694344:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c586d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694349:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265ca00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.694352:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c586d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694359:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c586d lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694365:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c586d lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694370:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.694373:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c586d lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694378:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c586d lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694654:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5874 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694664:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5874 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694670:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5874 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694677:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5874 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694683:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5874 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694688:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265c800 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201389.694692:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5874 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694698:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5874 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694705:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5874 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694710:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201389.694795:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5874 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.694803:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c800/0x4605c7c97e1c5874 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201389.694824:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442750080:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:1.0:1788201389.697063:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201389.700527:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d500 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442750336:192.168.201.130@tcp:36:rm.0 00000080:00200000:1.0:1788201389.700605:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x40e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201389.700615:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201389.700661:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x40e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201389.700671:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x40e:0x0](ffff9023449a63c8) 00000008:00010000:1.0:1788201389.700701:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x463:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5aff expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.700788:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.700805:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.700818:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00010000:00010000:1.0:1788201389.700832:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00000080:00200000:1.0:1788201389.700837:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.700841:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.700844:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.700850:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:1.0:1788201389.700853:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00010000:00010000:1.0:1788201389.700859:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 80 lru_type: 1 00000080:00200000:1.0:1788201389.700873:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.701108:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:1.0:1788201389.701112:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.701115:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-4, flags=8 00000080:00200008:1.0:1788201389.701153:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-4:[0x200000bd1:0x40f:0x0](ffff9023449b0908), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201389.701157:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x40f:0x0](ffff9023449b0908),name=t-4 00000002:00010000:1.0:1788201389.701169:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x40f:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.701175:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.701181:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.701186:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.701188:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x40f:0x0](ffff9023449b0908) 00000080:00010000:1.0:1788201389.701190:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x40f:0x0](ffff9023449b0908) for lock 0x4605c7c97e1c538f 00000080:00010000:1.0:1788201389.701193:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c538f from it ffffa0b181c3bcb8 00010000:00010000:1.0:1788201389.701195:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.701200:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201389.701218:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x40f:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788201389.701223:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x40f:0x0] 00000008:00010000:1.0:1788201389.701236:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788201389.701239:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023586d2d30, osc lock ffff90235090cbe0, flags 0x400001002 00010000:00010000:1.0:1788201389.701292:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 3/1,0 mode: --/PR res: [0x240000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.701302:0:17859:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 3/1,0 mode: --/PR res: [0x240000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788201389.701332:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1788201389.701340:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201389.701354:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442750464:192.168.201.130@tcp:101:rm.0 00010000:00010000:1.0:1788201389.704787:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.704819:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 6/2,0 mode: --/PR res: [0x240000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5b37 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.704834:0:2360:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5b37 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.704844:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5b37 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201389.704858:0:2360:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5b37 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788201389.704867:0:2360:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1788201389.704938:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5b37 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.704945:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5b37 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.704950:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5b37 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.704955:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5b37 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201389.704969:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442750464:192.168.201.130@tcp:101:rm.0 00000080:00200000:2.0:1788201389.705123:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x40f:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201389.705128:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235844d720 (ffff9023500e7998/0) at cl_lock_release():245 00000020:00010000:2.0:1788201389.705130:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235844d720 (ffff9023500e7998/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.705134:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023586d2d30 (ffff9023500e7ea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.705137:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235844d720 (ffff9023500e7998/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201389.705139:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023586d2d30 (ffff9023500e7998/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201389.705359:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x40f:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201389.705515:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201389.705534:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.705546:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 80 lru_type: 1 00010000:00010000:2.0:1788201389.705555:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 80 lru_type: 1 00000080:00200000:2.0:1788201389.705559:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.705566:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201389.705569:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201389.705573:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:2.0:1788201389.705575:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 80 lru_type: 1 00010000:00010000:2.0:1788201389.705580:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 81 lru_type: 1 00000080:00200000:2.0:1788201389.705586:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.705591:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:2.0:1788201389.705595:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.705600:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-4, flags=0 00000080:00200000:2.0:1788201389.705604:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:2.0:1788201389.705608:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-4, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:2.0:1788201389.705660:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201389.705675:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x40f:0x0]: losing layout lock 00010000:00010000:2.0:1788201389.705708:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201389.705802:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201389.705805:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731800/0x4605c7c97e1c538f lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x40f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d53dc expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201389.705833:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201389.705841:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234518ea00 x1875064442750720/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201389.705859:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518ea00 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442750720:192.168.201.130@tcp:36:rm.0 00000100:00100000:2.0:1788201389.706044:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3840 going to sleep for 11 seconds 00000100:00100000:2.0:1788201389.712589:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518ea00 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442750720:192.168.201.130@tcp:36:rm.0 00000080:00200000:2.0:1788201389.712666:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x40f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201389.712675:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201389.713010:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x40f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201389.713026:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x40f:0x0](ffff9023449b0908) 00000008:00010000:2.0:1788201389.713062:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x464:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5b37 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201389.713133:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201389.713233:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.713246:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 81 lru_type: 1 00010000:00010000:2.0:1788201389.713258:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 81 lru_type: 1 00000080:00200000:2.0:1788201389.713274:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.713281:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201389.713285:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201389.713295:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:2.0:1788201389.713300:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 81 lru_type: 1 00010000:00010000:2.0:1788201389.713310:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 82 lru_type: 1 00000080:00200000:2.0:1788201389.713327:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.713333:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:2.0:1788201389.713337:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.713340:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-5, flags=8 00000080:00200008:2.0:1788201389.713361:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-5:[0x200000bd1:0x410:0x0](ffff9023449b63c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201389.713474:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x410:0x0](ffff9023449b63c8),name=t-5 00000002:00010000:2.0:1788201389.713486:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x410:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.713492:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201389.713499:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201389.713503:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.713506:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x410:0x0](ffff9023449b63c8) 00000080:00010000:2.0:1788201389.713508:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x410:0x0](ffff9023449b63c8) for lock 0x4605c7c97e1c5396 00000080:00010000:2.0:1788201389.713511:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5396 from it ffffa0b181c3bcb8 00010000:00010000:2.0:1788201389.713512:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201389.713517:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201389.713536:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x410:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201389.713542:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x410:0x0] 00000008:00010000:2.0:1788201389.713559:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201389.713562:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398a30, osc lock ffff9023500dd8f0, flags 0x400001002 00010000:00010000:2.0:1788201389.713599:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.713605:0:17859:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201389.713665:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442750848:192.168.201.130@tcp:101:rm.0 00000020:00200000:2.0:1788201389.713827:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788201389.716343:0:2361:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.716364:0:2361:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5b5a expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.716455:0:2361:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5b5a expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.716465:0:2361:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5b5a expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201389.716476:0:2361:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5b5a expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201389.716484:0:2361:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788201389.716537:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5b5a expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.716543:0:2361:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5b5a expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.716548:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5b5a expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.716553:0:2361:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5b5a expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201389.716567:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442750848:192.168.201.130@tcp:101:rm.0 00000080:00200000:1.0:1788201389.716615:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x410:0x0] updating i_size 0 i_blocks 1 00000020:00010000:1.0:1788201389.716620:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc4000 (ffff902350b9df38/0) at cl_lock_release():245 00000020:00010000:1.0:1788201389.716622:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc4000 (ffff902350b9df38/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201389.716627:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398a30 (ffff902350b9d1b8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788201389.716629:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc4000 (ffff902350b9df38/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201389.716631:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398a30 (ffff902350b9df38/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201389.716659:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x410:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201389.716700:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.716715:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.716727:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 82 lru_type: 1 00010000:00010000:1.0:1788201389.716735:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 82 lru_type: 1 00000080:00200000:1.0:1788201389.716744:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.716750:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.716752:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.716756:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:1.0:1788201389.716759:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 82 lru_type: 1 00010000:00010000:1.0:1788201389.716764:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 83 lru_type: 1 00000080:00200000:1.0:1788201389.716774:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.716781:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:1.0:1788201389.716786:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.716792:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-5, flags=0 00000080:00200000:1.0:1788201389.716795:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201389.716800:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-5, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:1.0:1788201389.716863:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201389.717019:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x410:0x0]: losing layout lock 00010000:00010000:1.0:1788201389.717065:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201389.717071:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201389.717073:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731400/0x4605c7c97e1c5396 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x410:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d53ff expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201389.717093:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201389.717100:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234519f480 x1875064442751104/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201389.717118:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442751104:192.168.201.130@tcp:36:rm.0 00000100:00100000:1.0:1788201389.717261:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235312ad80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.724516:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442751104:192.168.201.130@tcp:36:rm.0 00000080:00200000:1.0:1788201389.724610:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x410:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201389.724626:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201389.724688:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x410:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201389.724702:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x410:0x0](ffff9023449b63c8) 00000008:00010000:1.0:1788201389.724806:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x464:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5b5a expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.724999:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.725017:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.725029:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 83 lru_type: 1 00010000:00010000:1.0:1788201389.725039:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 83 lru_type: 1 00000080:00200000:1.0:1788201389.725044:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.725048:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.725051:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.725057:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:1.0:1788201389.725060:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 83 lru_type: 1 00010000:00010000:1.0:1788201389.725075:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00000080:00200000:1.0:1788201389.725091:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.725097:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:1.0:1788201389.725102:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.725105:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-6, flags=8 00000080:00200008:1.0:1788201389.725246:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-6:[0x200000bd1:0x411:0x0](ffff90234497aa08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201389.725252:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x411:0x0](ffff90234497aa08),name=t-6 00000002:00010000:1.0:1788201389.725272:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x411:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.725281:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.725289:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.725294:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.725298:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x411:0x0](ffff90234497aa08) 00000080:00010000:1.0:1788201389.725301:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x411:0x0](ffff90234497aa08) for lock 0x4605c7c97e1c539d 00000080:00010000:1.0:1788201389.725305:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c539d from it ffffa0b181c3bcb8 00010000:00010000:1.0:1788201389.725308:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.725315:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201389.725348:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x411:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788201389.725355:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x411:0x0] 00000008:00010000:1.0:1788201389.725496:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788201389.725500:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023586d2d30, osc lock ffff90235090da20, flags 0x400001002 00010000:00010000:1.0:1788201389.725550:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.725559:0:17859:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201389.725633:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442751232:192.168.201.130@tcp:101:rm.0 00000020:00200000:1.0:1788201389.725791:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:0.0:1788201389.731017:0:2360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.731047:0:2360:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5b7d expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.731062:0:2360:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5b7d expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.731075:0:2360:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5b7d expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788201389.731094:0:2360:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5b7d expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788201389.731105:0:2360:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1788201389.731159:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5b7d expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.731169:0:2360:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5b7d expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.731176:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5b7d expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201389.731186:0:2360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5b7d expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788201389.731204:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442751232:192.168.201.130@tcp:101:rm.0 00000080:00200000:1.0:1788201389.731331:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x411:0x0] updating i_size 0 i_blocks 1 00000020:00010000:1.0:1788201389.731337:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235844d280 (ffff9023500e7ab8/0) at cl_lock_release():245 00000020:00010000:1.0:1788201389.731341:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235844d280 (ffff9023500e7ab8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201389.731346:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023586d2d30 (ffff9023500e7f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788201389.731349:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235844d280 (ffff9023500e7ab8/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201389.731352:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023586d2d30 (ffff9023500e7ab8/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201389.731485:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x411:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201389.731538:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.731555:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.731570:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00010000:00010000:1.0:1788201389.731582:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00000080:00200000:1.0:1788201389.731589:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.731596:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.731600:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.731607:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:1.0:1788201389.731610:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00010000:00010000:1.0:1788201389.731618:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00000080:00200000:1.0:1788201389.731628:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.731634:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:1.0:1788201389.731638:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.731644:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-6, flags=0 00000080:00200000:1.0:1788201389.731648:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201389.731652:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-6, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:1.0:1788201389.731696:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201389.731868:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x411:0x0]: losing layout lock 00010000:00010000:1.0:1788201389.734684:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201389.734701:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201389.734704:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685400/0x4605c7c97e1c539d lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x411:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5422 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201389.734734:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201389.734746:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234519e680 x1875064442751488/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201389.734774:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e680 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442751488:192.168.201.130@tcp:36:rm.0 00000100:00100000:1.0:1788201389.735027:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235312ad80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.739642:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e680 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442751488:192.168.201.130@tcp:36:rm.0 00000080:00200000:1.0:1788201389.739792:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x411:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201389.739806:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201389.739861:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x411:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201389.739997:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x411:0x0](ffff90234497aa08) 00000008:00010000:1.0:1788201389.740041:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x465:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5b7d expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.740125:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.740141:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.740153:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00010000:00010000:1.0:1788201389.740162:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00000080:00200000:1.0:1788201389.740167:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.740171:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.740174:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.740181:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:1.0:1788201389.740184:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00010000:00010000:1.0:1788201389.740194:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00000080:00200000:1.0:1788201389.740204:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.740209:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:1.0:1788201389.740212:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.740214:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-7, flags=8 00000080:00200008:1.0:1788201389.740240:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-7:[0x200000bd1:0x412:0x0](ffff9023449ac2c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201389.740245:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x412:0x0](ffff9023449ac2c8),name=t-7 00000002:00010000:1.0:1788201389.740259:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x412:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.740275:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.740284:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.740290:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.740294:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x412:0x0](ffff9023449ac2c8) 00000080:00010000:1.0:1788201389.740297:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x412:0x0](ffff9023449ac2c8) for lock 0x4605c7c97e1c53a4 00000080:00010000:1.0:1788201389.740300:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53a4 from it ffffa0b181c3bcb8 00010000:00010000:1.0:1788201389.740303:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.740311:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201389.740340:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x412:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788201389.740347:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x412:0x0] 00000008:00010000:1.0:1788201389.740367:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788201389.740371:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023586d2d30, osc lock ffff90235090da20, flags 0x400001002 00010000:00010000:1.0:1788201389.740432:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.740442:0:17859:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 3/1,0 mode: --/PR res: [0x280000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788201389.740475:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1788201389.740497:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442751616:192.168.201.130@tcp:101:rm.0 00010000:00010000:1.0:1788201389.743139:0:2359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.743173:0:2359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 6/2,0 mode: --/PR res: [0x280000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5ba0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.743187:0:2359:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5ba0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.743199:0:2359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5ba0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201389.743213:0:2359:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5ba0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788201389.743225:0:2359:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1788201389.743233:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5ba0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.743241:0:2359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5ba0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.743248:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5ba0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.743257:0:2359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5ba0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201389.743301:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442751616:192.168.201.130@tcp:101:rm.0 00000080:00200000:2.0:1788201389.743383:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x412:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201389.743395:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235844d280 (ffff9023500e7ab8/0) at cl_lock_release():245 00000020:00010000:2.0:1788201389.743398:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235844d280 (ffff9023500e7ab8/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.743403:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023586d2d30 (ffff9023500e7f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.743407:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235844d280 (ffff9023500e7ab8/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201389.743409:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023586d2d30 (ffff9023500e7ab8/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201389.743439:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x412:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201389.743486:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201389.743507:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.743521:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00010000:00010000:2.0:1788201389.743531:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00000080:00200000:2.0:1788201389.743538:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.743544:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201389.743548:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201389.743554:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:2.0:1788201389.743557:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00010000:00010000:2.0:1788201389.743565:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00000080:00200000:2.0:1788201389.743575:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.743581:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:2.0:1788201389.743587:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.743593:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-7, flags=0 00000080:00200000:2.0:1788201389.743596:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:2.0:1788201389.743601:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-7, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:2.0:1788201389.743662:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201389.743683:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x412:0x0]: losing layout lock 00010000:00010000:2.0:1788201389.743727:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201389.743734:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201389.743737:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685c00/0x4605c7c97e1c53a4 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x412:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5445 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201389.743762:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201389.743771:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234518f480 x1875064442751872/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201389.743793:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518f480 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442751872:192.168.201.130@tcp:36:rm.0 00000100:00100000:2.0:1788201389.743906:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.749707:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518f480 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442751872:192.168.201.130@tcp:36:rm.0 00000080:00200000:1.0:1788201389.749811:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x412:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201389.749822:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201389.749869:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x412:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201389.749952:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x412:0x0](ffff9023449ac2c8) 00000008:00010000:1.0:1788201389.749997:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x465:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5ba0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.750078:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.750095:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.750109:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00010000:00010000:1.0:1788201389.750122:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00000080:00200000:1.0:1788201389.750130:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.750135:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.750139:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.750146:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:1.0:1788201389.750150:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00010000:00010000:1.0:1788201389.750159:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 88 lru_type: 1 00000080:00200000:1.0:1788201389.750175:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.750181:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:1.0:1788201389.750185:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.750187:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-8, flags=8 00000080:00200008:1.0:1788201389.750204:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-8:[0x200000bd1:0x413:0x0](ffff9023449b7448), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201389.750207:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x413:0x0](ffff9023449b7448),name=t-8 00000002:00010000:1.0:1788201389.750217:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x413:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.750222:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.750228:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201389.750233:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.750235:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x413:0x0](ffff9023449b7448) 00000080:00010000:1.0:1788201389.750237:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x413:0x0](ffff9023449b7448) for lock 0x4605c7c97e1c53ab 00000080:00010000:1.0:1788201389.750239:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53ab from it ffffa0b181c3bcb8 00010000:00010000:1.0:1788201389.750241:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201389.750248:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201389.750285:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x413:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788201389.750292:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x413:0x0] 00000008:00010000:1.0:1788201389.750310:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788201389.750313:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023586d2630, osc lock ffff90235090db50, flags 0x400001002 00010000:00010000:1.0:1788201389.750365:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.750382:0:17859:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788201389.751615:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1788201389.751631:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201389.751652:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442752000:192.168.201.130@tcp:101:rm.0 00010000:00010000:1.0:1788201389.754548:0:2359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.754582:0:2359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5bc3 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.754597:0:2359:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5bc3 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.754609:0:2359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5bc3 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788201389.754624:0:2359:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5bc3 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788201389.754637:0:2359:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1788201389.754710:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5bc3 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.754729:0:2359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5bc3 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.754736:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5bc3 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201389.754744:0:2359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5bc3 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201389.754769:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442752000:192.168.201.130@tcp:101:rm.0 00000080:00200000:2.0:1788201389.754981:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x413:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201389.754987:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235844dbc0 (ffff9023500e7248/0) at cl_lock_release():245 00000020:00010000:2.0:1788201389.754991:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235844dbc0 (ffff9023500e7248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.754995:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023586d2630 (ffff9023500e7908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201389.755000:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235844dbc0 (ffff9023500e7248/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201389.755003:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023586d2630 (ffff9023500e7248/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201389.755042:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x413:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201389.755093:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201389.755115:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.755130:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 88 lru_type: 1 00010000:00010000:2.0:1788201389.755143:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 88 lru_type: 1 00000080:00200000:2.0:1788201389.755152:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.755159:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201389.755162:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201389.755167:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:2.0:1788201389.755171:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 88 lru_type: 1 00010000:00010000:2.0:1788201389.755179:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 89 lru_type: 1 00000080:00200000:2.0:1788201389.755190:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.755196:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:2.0:1788201389.755201:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.755207:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-8, flags=0 00000080:00200000:2.0:1788201389.755211:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:2.0:1788201389.755216:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-8, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:2.0:1788201389.755286:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201389.755312:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x413:0x0]: losing layout lock 00010000:00010000:2.0:1788201389.755357:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201389.755369:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201389.755372:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902343731c00/0x4605c7c97e1c53ab lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x413:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5468 expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201389.755489:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201389.755496:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234518fb80 x1875064442752256/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201389.755519:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518fb80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442752256:192.168.201.130@tcp:36:rm.0 00000100:00100000:2.0:1788201389.755636:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3240 going to sleep for 11 seconds 00000100:00100000:2.0:1788201389.759295:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518fb80 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442752256:192.168.201.130@tcp:36:rm.0 00000080:00200000:2.0:1788201389.759393:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x413:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201389.759405:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201389.759455:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x413:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201389.759466:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x413:0x0](ffff9023449b7448) 00000008:00010000:2.0:1788201389.759495:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x466:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5bc3 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201389.759561:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201389.759574:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.759583:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 89 lru_type: 1 00010000:00010000:2.0:1788201389.759592:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 89 lru_type: 1 00000080:00200000:2.0:1788201389.759601:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.759605:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201389.759607:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201389.759613:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bb58 00010000:00010000:2.0:1788201389.759615:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 89 lru_type: 1 00010000:00010000:2.0:1788201389.759621:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 90 lru_type: 1 00000080:00200000:2.0:1788201389.759631:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.759635:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:2.0:1788201389.759639:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:2.0:1788201389.759642:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-9, flags=8 00000080:00200008:2.0:1788201389.759659:0:17859:0:(file.c:6283:ll_getattr_dentry()) START file t-9:[0x200000bd1:0x414:0x0](ffff9023449a7448), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201389.759662:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x414:0x0](ffff9023449a7448),name=t-9 00000002:00010000:2.0:1788201389.759670:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x414:0x0]) in obj [0x200000bd1:0x40a:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201389.759675:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201389.759680:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201389.759684:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000bd1:0x40a:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201389.759686:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x414:0x0](ffff9023449a7448) 00000080:00010000:2.0:1788201389.759687:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x414:0x0](ffff9023449a7448) for lock 0x4605c7c97e1c53b2 00000080:00010000:2.0:1788201389.759690:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c53b2 from it ffffa0b181c3bcb8 00010000:00010000:2.0:1788201389.759692:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201389.759696:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201389.759721:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x414:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201389.759728:0:17859:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x414:0x0] 00000008:00010000:2.0:1788201389.759746:0:17859:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201389.759749:0:17859:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902353398030, osc lock ffff9023500dd560, flags 0x400001002 00010000:00010000:2.0:1788201389.759793:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 3/1,0 mode: --/PR res: [0x280000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.759802:0:17859:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 3/1,0 mode: --/PR res: [0x280000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201389.759833:0:17859:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1788201389.759847:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201389.759860:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442752384:192.168.201.130@tcp:101:rm.0 00010000:00010000:2.0:1788201389.763665:0:2361:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.763693:0:2361:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 6/2,0 mode: --/PR res: [0x280000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5be6 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.763708:0:2361:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5be6 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.763846:0:2361:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5be6 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1788201389.763863:0:2361:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 5/2,0 mode: PR/PR res: [0x280000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5be6 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1788201389.763916:0:2361:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:2.0:1788201389.763932:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 6/2,0 mode: PR/PR res: [0x280000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5be6 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.763941:0:2361:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5be6 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.763949:0:2361:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d5be6 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201389.763957:0:2361:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5be6 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201389.763976:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442752384:192.168.201.130@tcp:101:rm.0 00000080:00200000:1.0:1788201389.764064:0:17859:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x414:0x0] updating i_size 0 i_blocks 1 00000020:00010000:1.0:1788201389.764071:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc4940 (ffff902350b9d7e8/0) at cl_lock_release():245 00000020:00010000:1.0:1788201389.764074:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc4940 (ffff902350b9d7e8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201389.764079:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902353398030 (ffff902350b9de18/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788201389.764082:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc4940 (ffff902350b9d7e8/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201389.764085:0:17859:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902353398030 (ffff902350b9d7e8/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201389.764117:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x414:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201389.764168:0:17859:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201389.764191:0:17859:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201389.764206:0:17859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 90 lru_type: 1 00010000:00010000:1.0:1788201389.764218:0:17859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 90 lru_type: 1 00000080:00200000:1.0:1788201389.764225:0:17859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201389.764231:0:17859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201389.764235:0:17859:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:1.0:1788201389.764241:0:17859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181c3bc10 00010000:00010000:1.0:1788201389.764244:0:17859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 90 lru_type: 1 00010000:00010000:1.0:1788201389.764252:0:17859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00000080:00200000:1.0:1788201389.764276:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.764284:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d103.replay-single, flags=24 00000080:00200000:1.0:1788201389.764289:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 1 00000080:00200000:1.0:1788201389.764296:0:17859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t-9, flags=0 00000080:00200000:1.0:1788201389.764300:0:17859:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x40a:0x0](ffff902344979148), inode mode 41ed mask 3 00000080:00200000:1.0:1788201389.764305:0:17859:0:(namei.c:2418:ll_unlink()) VFS Op:name=t-9, dir=[0x200000bd1:0x40a:0x0](ffff902344979148) 00010000:00010000:1.0:1788201389.764365:0:17859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201389.764530:0:17859:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x414:0x0]: losing layout lock 00010000:00010000:1.0:1788201389.764572:0:17859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201389.764602:0:17859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201389.764606:0:17859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345803200/0x4605c7c97e1c53b2 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x414:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d548b expref: -99 pid: 17857 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201389.764631:0:17859:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201389.764639:0:17859:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234519c380 x1875064442752640/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201389.764660:0:17859:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442752640:192.168.201.130@tcp:36:rm.0 00000100:00100000:1.0:1788201389.764996:0:17859:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235312ad80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201389.776741:0:17859:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:17859:1875064442752640:192.168.201.130@tcp:36:rm.0 00000080:00200000:1.0:1788201389.776832:0:17859:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x414:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201389.776844:0:17859:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201389.777592:0:17859:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x414:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201389.777609:0:17859:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x414:0x0](ffff9023449a7448) 00000008:00010000:1.0:1788201389.777647:0:17859:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 0 0 0 by oinfo size 0 blocks 1 [cma]time 0 0 0 ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 1/0,0 mode: PR/PR res: [0x280000400:0x466:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d5be6 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201390.723530:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201390.986701:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442752768:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201390.986831:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442753024:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201390.986869:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:0.0:1788201390.986910:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442752896:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201390.986939:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442753152:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201390.992326:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442752896:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201390.992345:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442753152:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201390.992367:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442752768:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201390.992378:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442753024:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201391.478654:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201391.478706:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201391.478724:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201391.478744:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201391.478925:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458266496 00000100:00100000:3.0:1788201391.478934:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458266496 00000100:00100000:3.0:1788201391.478974:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458266752 00000100:00100000:3.0:1788201391.478978:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458266752 00000100:00100000:3.0:1788201391.478989:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458266880 00000100:00100000:3.0:1788201391.478991:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458266880 00000100:00100000:3.0:1788201391.479000:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458267008 00000100:00100000:3.0:1788201391.479002:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458267008 00000100:00100000:3.0:1788201391.479018:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1339 00000100:00100000:3.0:1788201391.479027:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+6:13449:x1875064458266496:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201391.479046:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5771 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201391.479195:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201391.479203:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+6:13449:x1875064458266496:12345-192.168.201.130@tcp:104: Request processed in 177us (558us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201391.479288:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1339 00000100:00100000:3.0:1788201391.479309:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1340 00000100:00100000:3.0:1788201391.479316:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+5:13439:x1875064458266752:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201391.479326:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1788201391.479590:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1341 00000100:00100000:2.0:1788201391.479599:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:13450:x1875064458266880:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201391.479609:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000010000 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788201391.479660:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to priority blp list, 3 blwis in pool 00000100:00100000:3.0:1788201391.479667:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+5:13439:x1875064458266752:12345-192.168.201.130@tcp:104: Request processed in 352us (961us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201391.479674:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1340 00000100:00100000:3.0:1788201391.479688:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1342 00000100:00100000:3.0:1788201391.479695:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13445:x1875064458267008:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201391.479705:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5794 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.482643:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 3 total in blp. (3 blwis in pool) 00010000:00010000:1.0:1788201391.482657:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5771 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.482672:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345106200 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:1.0:1788201391.482680:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d5771 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.482790:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5771 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.482798:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201391.482825:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5771 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.482834:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009763e6ce/0x4605c7c97e1c5453 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5771 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.482856:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:1.0:1788201391.482860:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201391.482872:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839c200 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:1.0:1788201391.482898:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201391.482929:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201391.482936:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201391.482942:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201391.482948:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a7df05dc/0x4605c7c97e1c5127 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x442:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d49b0 expref: -99 pid: 16412 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201391.482961:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:1.0:1788201391.482964:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800420000010000 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201391.482971:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351684000 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:1.0:1788201391.482974:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800428400010000 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1788201391.483005:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x408:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201391.483016:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x408:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201391.483044:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400010020 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201391.483051:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201391.483056:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400010020 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201391.483063:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f095939/0x4605c7c97e1c5143 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x444:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804c69400010020 nid: local remote: 0x9383f985bc4d4a58 expref: -99 pid: 16416 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201391.483075:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201391.483079:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5794 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.485912:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902343731e00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:1.0:1788201391.485923:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d5794 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.486029:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 2/0,0 mode: --/PR res: [0x280000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5794 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.486037:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201391.486049:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 1/0,0 mode: --/PR res: [0x280000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5794 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.486057:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000004954c984/0x4605c7c97e1c545a lrc: 0/0,0 mode: --/PR res: [0x280000400:0x462:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5794 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.486428:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201391.486430:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201391.486434:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201391.486463:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442753408:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201391.486623:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442753664:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201391.486648:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788201391.486654:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442753280:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201391.486674:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442753536:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201391.488286:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/0 locks to priority blp list, 0 blwis in pool 00000100:00100000:2.0:1788201391.488305:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13450:x1875064458266880:12345-192.168.201.130@tcp:104: Request processed in 8703us (9573us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201391.488317:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1341 00010000:00010000:1.0:1788201391.488990:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201391.494486:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/0 locks to priority blp list, 0 blwis in pool 00000100:00100000:3.0:1788201391.494505:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13445:x1875064458267008:12345-192.168.201.130@tcp:104: Request processed in 14803us (15752us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201391.494516:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1342 00000100:00100000:2.0:1788201391.495274:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201391.495383:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00010000:00010000:0.0:1788201391.495611:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201391.495630:0:5589:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458267520 00000100:00100000:0.0:1788201391.495639:0:5589:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458267520 00000100:00100000:0.0:1788201391.495685:0:5589:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 200 00000100:00100000:0.0:1788201391.495696:0:5589:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:13443:x1875064458267520:12345-192.168.201.130@tcp:104: 00010000:00010000:0.0:1788201391.495713:0:5589:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5820 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.495786:0:5589:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788201391.495793:0:5589:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:LOV_OSC_UUID+4:13443:x1875064458267520:12345-192.168.201.130@tcp:104: Request processed in 100us (410us total) trans 0 rc 0/0 00000100:00100000:0.0:1788201391.495800:0:5589:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 200 00010000:00010000:0.0:1788201391.495833:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201391.495840:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5820 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.495849:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839ce00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:0.0:1788201391.495856:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d5820 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.495975:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5820 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.495984:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201391.496004:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5820 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.496012:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000cf1eab0f/0x4605c7c97e1c5476 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5820 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.496034:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201391.496036:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201391.496038:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201391.496060:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442753792:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201391.496472:0:11886:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1788201391.496487:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458267264 00000100:00100000:3.0:1788201391.496492:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458267264 00000100:00100000:3.0:1788201391.496523:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1343 00000100:00100000:3.0:1788201391.496530:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023584a7450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13447:x1875064458267264:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201391.496540:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d57da expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201391.496583:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201391.496589:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023584a7450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13447:x1875064458267264:12345-192.168.201.130@tcp:104: Request processed in 60us (1321us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201391.496596:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1343 00010000:00010000:0.0:1788201391.498028:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201391.498040:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d57da expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.498053:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902343731600 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:0.0:1788201391.498059:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d57da expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.498332:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d57da expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.498406:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201391.498431:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d57da expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.498437:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000ca73d97f/0x4605c7c97e1c5468 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d57da expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.498458:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201391.498460:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201391.498462:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201391.498483:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442753920:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201391.502954:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201391.503121:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201391.503143:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201391.503170:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458268288 00000100:00100000:2.0:1788201391.503178:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458268288 00000100:00100000:2.0:1788201391.503340:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458268416 00000100:00100000:2.0:1788201391.503344:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458268416 00000100:00100000:2.0:1788201391.503357:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458268672 00000100:00100000:2.0:1788201391.503359:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458268672 00000100:00100000:2.0:1788201391.503373:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1344 00000100:00100000:2.0:1788201391.503383:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+6:6703:x1875064458268288:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201391.503402:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503478:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201391.503486:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+6:6703:x1875064458268288:12345-192.168.201.130@tcp:104: Request processed in 104us (544us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201391.503494:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1344 00000100:00100000:2.0:1788201391.503510:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1345 00000100:00100000:2.0:1788201391.503516:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13446:x1875064458268416:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201391.503525:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503553:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:2.0:1788201391.503559:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13446:x1875064458268416:12345-192.168.201.130@tcp:104: Request processed in 43us (438us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201391.503565:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1345 00000100:00100000:2.0:1788201391.503575:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1346 00000100:00100000:2.0:1788201391.503581:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234518f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13448:x1875064458268672:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201391.503589:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503614:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to priority blp list, 3 blwis in pool 00000100:00100000:2.0:1788201391.503619:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234518f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13448:x1875064458268672:12345-192.168.201.130@tcp:104: Request processed in 38us (475us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201391.503625:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1346 00010000:00010000:2.0:1788201391.503639:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:2.0:1788201391.503644:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503653:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345802c00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201391.503659:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503737:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503745:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201391.503765:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503774:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000dcc47ed8/0x4605c7c97e1c5492 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5858 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503795:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1788201391.503799:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503806:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345107000 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201391.503810:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503842:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 2/0,0 mode: --/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503849:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201391.503856:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 1/0,0 mode: --/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503869:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000d4c35712/0x4605c7c97e1c54bc lrc: 0/0,0 mode: --/PR res: [0x280000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58cf expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503905:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201391.503908:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503915:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345802400 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:2.0:1788201391.503917:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503949:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 2/0,0 mode: --/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503957:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201391.503962:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 1/0,0 mode: --/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503969:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000003cfdd801/0x4605c7c97e1c54ae lrc: 0/0,0 mode: --/PR res: [0x280000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d586d expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.503982:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201391.503983:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201391.503986:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201391.504001:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234518ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442754048:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201391.504046:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442754304:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201391.504067:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201391.504073:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442754176:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201391.504355:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201391.504400:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201391.504492:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458268800 00000100:00100000:3.0:1788201391.504497:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458268800 00000100:00100000:3.0:1788201391.504517:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458268544 00000100:00100000:3.0:1788201391.504520:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458268544 00000100:00100000:3.0:1788201391.504532:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1347 00000100:00100000:3.0:1788201391.504539:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902352689180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:6706:x1875064458268800:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201391.504549:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201391.504602:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201391.504608:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902352689180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:6706:x1875064458268800:12345-192.168.201.130@tcp:104: Request processed in 69us (254us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201391.504678:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1347 00000100:00100000:3.0:1788201391.504693:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1348 00000100:00100000:3.0:1788201391.504699:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90235268b480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13451:x1875064458268544:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201391.504707:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201391.504746:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1788201391.504751:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90235268b480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13451:x1875064458268544:12345-192.168.201.130@tcp:104: Request processed in 53us (351us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201391.504757:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1348 00010000:00010000:3.0:1788201391.504784:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1788201391.504789:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201391.504795:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345106600 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201391.504799:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201391.504946:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201391.504953:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201391.504960:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201391.504967:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000018845e0a/0x4605c7c97e1c5484 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d584a expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201391.504981:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201391.504984:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201391.504989:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345803a00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:3.0:1788201391.504993:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201391.505020:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201391.505027:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201391.505031:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201391.505043:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000091077253/0x4605c7c97e1c54a0 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5866 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201391.505052:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201391.505053:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201391.505054:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201391.505079:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442754432:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201391.505336:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201391.505345:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442754560:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788201391.505437:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201391.506594:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442753408:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788201391.506622:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201391.507187:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201391.512479:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442753792:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201391.512569:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442753280:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201391.512579:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442753536:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201391.512588:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442753664:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201391.514354:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442753920:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201391.514444:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234518ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442754048:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201391.519653:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201391.520106:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458269056 00000100:00100000:2.0:1788201391.520116:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458269056 00000100:00100000:2.0:1788201391.520185:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1349 00000100:00100000:2.0:1788201391.520195:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902352688380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13440:x1875064458269056:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201391.520213:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.520573:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201391.520581:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902352688380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13440:x1875064458269056:12345-192.168.201.130@tcp:104: Request processed in 387us (938us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201391.520590:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1349 00010000:00010000:0.0:1788201391.520615:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201391.520624:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.520634:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865b200 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:0.0:1788201391.520640:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.520718:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 2/0,0 mode: --/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.520725:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201391.520747:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 1/0,0 mode: --/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.520755:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c554f lrc: 0/0,0 mode: --/PR res: [0x280000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58e4 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.520789:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201391.520792:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201391.520794:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201391.520802:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201391.520817:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442754688:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201391.526386:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442754560:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201391.526402:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442754432:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201391.526443:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442754304:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201391.530524:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442754176:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201391.533823:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:0.0:1788201391.538969:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442754688:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201391.540045:0:5591:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1788201391.540067:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458269312 00000100:00100000:2.0:1788201391.540073:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458269312 00000100:00100000:2.0:1788201391.540140:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1350 00000100:00100000:2.0:1788201391.540149:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023584a4c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13438:x1875064458269312:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201391.540167:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201391.540259:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201391.540266:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023584a4c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13438:x1875064458269312:12345-192.168.201.130@tcp:104: Request processed in 119us (6453us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201391.540273:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1350 00010000:00010000:0.0:1788201391.540898:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201391.540908:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.540919:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865be00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:0.0:1788201391.540926:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.541018:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 2/0,0 mode: --/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.541025:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201391.541058:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 1/0,0 mode: --/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.541067:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000044ed2b68/0x4605c7c97e1c555d lrc: 0/0,0 mode: --/PR res: [0x280000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58eb expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.541091:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201391.541093:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201391.541095:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201391.541159:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442754816:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201391.542548:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201391.542618:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458270080 00000100:00100000:3.0:1788201391.542626:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458270080 00000100:00100000:3.0:1788201391.542674:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1351 00000100:00100000:3.0:1788201391.543090:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234598dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13444:x1875064458270080:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201391.543108:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5ba0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201391.543196:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201391.543203:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234598dc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13444:x1875064458270080:12345-192.168.201.130@tcp:104: Request processed in 521us (664us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201391.543211:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1351 00010000:00010000:0.0:1788201391.543392:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201391.543402:0:5598:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5ba0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.543411:0:5598:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265ce00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:0.0:1788201391.543419:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d5ba0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.543502:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5ba0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.543509:0:5598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201391.543548:0:5598:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5ba0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.543557:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 000000001643e25d/0x4605c7c97e1c5890 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5ba0 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788201391.543590:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201391.543606:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442754944:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788201391.543669:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201391.543671:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201391.543673:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201391.549770:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442754816:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201391.553304:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201391.553334:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201391.553502:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:0.0:1788201391.555746:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442754944:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201391.556734:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201391.556833:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458270208 00000100:00100000:3.0:1788201391.556842:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458270208 00000100:00100000:3.0:1788201391.556857:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458270336 00000100:00100000:3.0:1788201391.556860:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458270336 00000100:00100000:3.0:1788201391.557005:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458270464 00000100:00100000:3.0:1788201391.557008:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458270464 00000100:00100000:3.0:1788201391.557025:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1352 00000100:00100000:3.0:1788201391.557033:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234598ea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+6:10484:x1875064458270336:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201391.557052:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5aff expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201391.557182:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201391.557189:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234598ea00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+6:10484:x1875064458270336:12345-192.168.201.130@tcp:104: Request processed in 156us (3854us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201391.557196:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1352 00000100:00100000:3.0:1788201391.557214:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1353 00000100:00100000:3.0:1788201391.557291:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234598ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+5:13442:x1875064458270464:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201391.557300:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5b5a expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201391.557414:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1788201391.557421:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234598ce00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+5:13442:x1875064458270464:12345-192.168.201.130@tcp:104: Request processed in 200us (3920us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201391.557427:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1353 00000100:00100000:3.0:1788201391.557464:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1354 00000100:00100000:3.0:1788201391.557471:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234598ed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13443:x1875064458270208:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201391.557480:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.557554:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:0.0:1788201391.557561:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5aff expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.557570:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865b400 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:0.0:1788201391.557576:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d5aff expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.557647:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 2/0,0 mode: --/PR res: [0x280000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5aff expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.557654:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201391.557678:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 1/0,0 mode: --/PR res: [0x280000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5aff expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.557687:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c3a89c40/0x4605c7c97e1c582e lrc: 0/0,0 mode: --/PR res: [0x280000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5aff expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201391.558200:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458270592 00000100:00100000:2.0:1788201391.558205:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458270592 00000100:00100000:2.0:1788201391.558242:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1355 00000100:00100000:2.0:1788201391.558250:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234598df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13445:x1875064458270592:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201391.558259:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5be6 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201391.558299:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/3 locks to priority blp list, 3 blwis in pool 00000100:00100000:2.0:1788201391.558305:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234598df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13445:x1875064458270592:12345-192.168.201.130@tcp:104: Request processed in 57us (1576us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201391.558312:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1355 00010000:00010000:0.0:1788201391.559512:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:0.0:1788201391.559521:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5b5a expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.559532:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345107200 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:0.0:1788201391.559539:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d5b5a expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.559614:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 2/0,0 mode: --/PR res: [0x280000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5b5a expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.559621:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201391.559634:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5b5a expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.559641:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000c0d58b15/0x4605c7c97e1c5882 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5b5a expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.559658:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1788201391.559662:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.559673:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342e70800 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:0.0:1788201391.559676:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.561478:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 2/0,0 mode: --/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.561496:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201391.561510:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 1/0,0 mode: --/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.561518:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 0000000086fd2077/0x4605c7c97e1c556b lrc: 0/0,0 mode: --/PR res: [0x280000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58f2 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.561543:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201391.561547:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5be6 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.561555:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345107c00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:0.0:1788201391.561560:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 3/0,0 mode: PR/PR res: [0x280000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d5be6 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.561618:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 2/0,0 mode: --/PR res: [0x280000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5be6 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.561632:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201391.561637:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 1/0,0 mode: --/PR res: [0x280000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5be6 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.561644:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff902350a11000 lock: 00000000b82790c9/0x4605c7c97e1c589e lrc: 0/0,0 mode: --/PR res: [0x280000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5be6 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201391.561658:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201391.561660:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201391.561663:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201391.561675:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201391.561692:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442755072:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201391.564427:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442755328:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201391.564491:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1788201391.564498:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442755200:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201391.564516:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442755456:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788201391.564549:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201391.565405:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/0 locks to priority blp list, 0 blwis in pool 00000100:00100000:3.0:1788201391.565420:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234598ed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13443:x1875064458270208:12345-192.168.201.130@tcp:104: Request processed in 7945us (12122us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201391.565431:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1354 00010000:00010000:1.0:1788201391.566049:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201391.574624:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442755072:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201391.576427:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442755328:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201391.576511:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442755200:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201391.576520:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442755456:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201391.612072:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201391.612162:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201391.612274:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201391.612308:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:0.0:1788201391.612395:0:5588:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:0.0:1788201391.612427:0:5588:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458272640 00000100:00100000:0.0:1788201391.612435:0:5588:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458272640 00000100:00100000:0.0:1788201391.612481:0:5588:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458272896 00000100:00100000:0.0:1788201391.612484:0:5588:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458272896 00000100:00100000:0.0:1788201391.612500:0:5588:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 201 00000100:00100000:0.0:1788201391.612508:0:5588:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234598fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+5:6706:x1875064458272640:12345-192.168.201.130@tcp:104: 00010000:00010000:0.0:1788201391.612524:0:5588:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.612589:0:5588:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788201391.612595:0:5588:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234598fb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+5:6706:x1875064458272640:12345-192.168.201.130@tcp:104: Request processed in 88us (321us total) trans 0 rc 0/0 00000100:00100000:0.0:1788201391.612602:0:5588:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 201 00000100:00100000:0.0:1788201391.612619:0:5588:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 202 00000100:00100000:0.0:1788201391.612626:0:5588:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902345930450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:13450:x1875064458272896:12345-192.168.201.130@tcp:104: 00010000:00010000:0.0:1788201391.612634:0:5588:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.612663:0:5588:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:0.0:1788201391.612669:0:5588:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902345930450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:13450:x1875064458272896:12345-192.168.201.130@tcp:104: Request processed in 44us (362us total) trans 0 rc 0/0 00000100:00100000:0.0:1788201391.612675:0:5588:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 202 00010000:00010000:0.0:1788201391.612724:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1788201391.612731:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.612740:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345106e00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:0.0:1788201391.612746:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.612829:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.612836:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201391.612857:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.612865:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cca786fe/0x4605c7c97e1c5499 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x46b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d585f expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.612923:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201391.612927:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.612934:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345802800 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:0.0:1788201391.612938:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.612973:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.612981:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201391.612988:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.613001:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e4e38a3/0x4605c7c97e1c54a7 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x46c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d587b expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201391.613013:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201391.613015:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201391.613017:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201391.613033:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442755712:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201391.613078:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201391.613084:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442755584:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788201391.613113:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201391.613289:0:5591:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1788201391.613303:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458272512 00000100:00100000:3.0:1788201391.613307:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458272512 00000100:00100000:3.0:1788201391.613390:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458272768 00000100:00100000:3.0:1788201391.613394:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458272768 00000100:00100000:3.0:1788201391.613412:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1356 00000100:00100000:3.0:1788201391.613419:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234598f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13446:x1875064458272512:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201391.613432:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201391.613474:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201391.613479:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234598f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13446:x1875064458272512:12345-192.168.201.130@tcp:104: Request processed in 62us (1419us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201391.613485:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1356 00000100:00100000:3.0:1788201391.613500:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1357 00000100:00100000:3.0:1788201391.613506:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902345193050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13440:x1875064458272768:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201391.613514:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201391.613583:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:3.0:1788201391.613589:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902345193050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13440:x1875064458272768:12345-192.168.201.130@tcp:104: Request processed in 83us (1429us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201391.613595:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1357 00000100:00100000:3.0:1788201391.613660:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201391.613722:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201391.613750:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201391.613783:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201391.614068:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458273152 00000100:00100000:3.0:1788201391.614074:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458273152 00000100:00100000:3.0:1788201391.614089:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458273280 00000100:00100000:3.0:1788201391.614092:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458273280 00000100:00100000:3.0:1788201391.614100:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458273408 00000100:00100000:3.0:1788201391.614102:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458273408 00000100:00100000:3.0:1788201391.614114:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1358 00000100:00100000:3.0:1788201391.614120:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902352689c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+6:13447:x1875064458273152:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201391.614129:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d59af expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201391.614202:0:11886:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 0 new 8192-byte reqbufs (1/1 left), rc = 0 00000100:00100000:2.0:1788201391.614211:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1359 00000100:00100000:2.0:1788201391.614283:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90235268a680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+6:13439:x1875064458273280:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201391.614292:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d57b7 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788201391.614406:0:5588:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458273024 00000100:00100000:0.0:1788201391.614410:0:5588:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458273024 00000100:00100000:0.0:1788201391.614436:0:5588:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 203 00000100:00100000:0.0:1788201391.614442:0:5588:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902352689f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+7:6705:x1875064458273024:12345-192.168.201.130@tcp:104: 00010000:00010000:0.0:1788201391.614450:0:5588:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d57fd expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.614769:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 4 total in blp. (4 blwis in pool) 00010000:00010000:1.0:1788201391.614777:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201391.614792:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345106c00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:1.0:1788201391.614798:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201391.614887:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 2/0,0 mode: --/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201391.614896:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201391.614916:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 1/0,0 mode: --/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201391.614925:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cdd7de4/0x4605c7c97e1c548b lrc: 0/0,0 mode: --/PR res: [0x240000400:0x46a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5851 expref: -99 pid: 17861 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:1.0:1788201391.614960:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442755840:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201391.615021:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 3 total in blp. (3 blwis in pool) 00010000:00010000:1.0:1788201391.615026:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201391.615033:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902342ff6a00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:1.0:1788201391.615036:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201391.615070:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201391.615078:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201391.615090:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201391.615097:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091c950c7/0x4605c7c97e1c54b5 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x46d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5874 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:1.0:1788201391.616491:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442755968:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201391.616581:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 2 total in blp. (2 blwis in pool) 00010000:00010000:1.0:1788201391.616587:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d59af expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.616597:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265c000 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:1.0:1788201391.616603:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d59af expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.616672:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d59af expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.616680:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201391.616705:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d59af expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.616723:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ca578aeb/0x4605c7c97e1c5820 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x463:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d59af expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201391.616756:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442756096:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201391.616793:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:1.0:1788201391.616797:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d57b7 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.616805:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345106a00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:1.0:1788201391.616808:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d57b7 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.616845:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d57b7 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.616852:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201391.616863:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d57b7 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.616871:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d238f9a9/0x4605c7c97e1c5461 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x467:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d57b7 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201391.616911:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442756224:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201391.616946:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201391.616951:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d57fd expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.616963:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902343731a00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:1.0:1788201391.616966:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d57fd expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.617001:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 2/0,0 mode: --/PR res: [0x240000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d57fd expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.617008:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201391.617019:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 1/0,0 mode: --/PR res: [0x240000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d57fd expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.617026:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003025e09b/0x4605c7c97e1c546f lrc: 0/0,0 mode: --/PR res: [0x240000400:0x468:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d57fd expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788201391.617046:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442756352:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201391.617078:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201391.617081:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201391.617084:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201391.617266:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201391.617770:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201391.618041:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/0 locks to priority blp list, 0 blwis in pool 00000100:00100000:3.0:1788201391.618055:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902352689c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+7:13447:x1875064458273152:12345-192.168.201.130@tcp:104: Request processed in 3930us (4328us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201391.618065:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1358 00000100:00100000:3.0:1788201391.618097:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1360 00000100:00100000:3.0:1788201391.618108:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90235268aa00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+6:6703:x1875064458273408:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201391.618132:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5843 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201391.618202:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201391.618209:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90235268aa00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+6:6703:x1875064458273408:12345-192.168.201.130@tcp:104: Request processed in 104us (4425us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201391.618215:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1360 00000100:00100000:1.0:1788201391.618484:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442755584:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201391.618511:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201391.618518:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5843 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.618532:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90234839d400 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:1.0:1788201391.618537:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d5843 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.618601:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 2/0,0 mode: --/PR res: [0x240000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5843 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.618608:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201391.618625:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 1/0,0 mode: --/PR res: [0x240000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5843 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.618632:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2a057aa/0x4605c7c97e1c547d lrc: 0/0,0 mode: --/PR res: [0x240000400:0x469:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5843 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201391.618651:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201391.618653:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201391.618655:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201391.618665:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442756480:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201391.618758:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/0 locks to priority blp list, 0 blwis in pool 00000100:00100000:2.0:1788201391.618766:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90235268a680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+5:13439:x1875064458273280:12345-192.168.201.130@tcp:104: Request processed in 4547us (5014us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201391.618772:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1359 00010000:00010000:2.0:1788201391.618798:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201391.619076:0:5588:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/0 locks to priority blp list, 0 blwis in pool 00000100:00100000:0.0:1788201391.619085:0:5588:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902352689f80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:LOV_OSC_UUID+4:6705:x1875064458273024:12345-192.168.201.130@tcp:104: Request processed in 4641us (5425us total) trans 0 rc 0/0 00000100:00100000:0.0:1788201391.619092:0:5588:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 203 00010000:00010000:2.0:1788201391.619564:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201391.624068:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442755712:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201391.630186:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442756480:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201391.630202:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442756224:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201391.630210:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442755968:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201391.630535:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442756352:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201391.630544:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442756096:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201391.630555:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442755840:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201391.745278:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201392.769258:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201393.794062:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201394.818807:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201395.847080:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201396.098872:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442756608:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201396.098873:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442756736:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201396.098999:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442756864:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201396.099105:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442756992:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201396.105748:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442756992:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201396.106520:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442756864:192.168.201.130@tcp:400:kworker.0 00010000:00010000:1.0:1788201396.865129:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201397.889225:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201398.919098:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201399.937138:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201400.964398:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201401.217737:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442757120:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201401.217845:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442757376:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201401.217918:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788201401.217926:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442757248:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201401.218049:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442757504:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201401.221483:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442757376:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201401.223963:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442757504:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201401.994185:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201403.009087:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201404.034078:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201405.062071:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201406.082034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201406.337276:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442757632:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201406.337386:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442757888:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201406.337418:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201406.337424:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442757760:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201406.337441:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442758016:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201406.358359:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442757888:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201406.358403:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442758016:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201407.106976:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201408.129395:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201409.153664:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201410.177072:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201410.945373:0:17241:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c58a5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201410.945392:0:17241:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c58a5 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x0 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201410.945570:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442758144:192.168.201.130@tcp:63:ll_sa.0 00000100:02020000:1.0:1788201410.949709:0:2360:0:(client.c:1492:ptlrpc_check_status()) lustre-MDT0000-mdc-ffff902350a11000: operation mds_batch to node 192.168.201.130@tcp failed: rc = -107 00000100:00080000:1.0:1788201410.957229:0:2360:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff902350a11000 of lustre-MDT0000_UUID@192.168.201.130@tcp abruptly disconnected: reconnecting 00000100:00080000:1.0:1788201410.957237:0:2360:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:1.0:1788201410.957242:0:2360:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff902350a11000: Connection to lustre-MDT0000 (at 192.168.201.130@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1788201410.974344:0:2360:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201410.974361:0:2360:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 647 00000100:00080000:1.0:1788201410.974376:0:2360:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201410.974389:0:2360:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201410.974424:0:2360:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442758272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201410.974599:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442758272:192.168.201.130@tcp:38:ptlrpcd_00_01.0 00010000:00010000:2.0:1788201411.201247:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201411.457277:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201411.457284:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201411.457305:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:02000000:2.0:1788201411.457309:0:11804:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff902350a11000: disconnect after 20s idle 00000100:00100000:3.0:1788201411.457939:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442758400:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201411.458062:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201411.458070:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442758528:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201411.464644:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442758528:192.168.201.130@tcp:400:kworker.0 00000100:00080000:3.0:1788201411.464663:0:2362:0:(recover.c:205:ptlrpc_request_handle_notconn()) import MGC192.168.201.130@tcp of MGS@192.168.201.130@tcp abruptly disconnected: reconnecting 00000100:00080000:3.0:1788201411.464669:0:2362:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from FULL to DISCONN 00000100:02020000:3.0:1788201411.464672:0:2362:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail 00000100:00100000:0.0:1788201411.468244:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442758656:192.168.201.130@tcp:9:kworker.0 00000100:00080000:3.0:1788201411.492782:0:2362:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.201.130@tcp for MGC192.168.201.130@tcp not replayable, auto-deactivating 00000100:00080000:3.0:1788201411.492787:0:2362:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:3.0:1788201411.492797:0:2362:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff90234598c000 x1875064442756608/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201412 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201411.492811:0:2362:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff9023450ba300 x1875064442757120/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201417 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201411.492836:0:2362:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff90234598f800 x1875064442757632/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201422 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201411.492844:0:2362:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff90234598ca80 x1875064442758400/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201427 ref 1 fl Rpc:RNQU/200/0 rc 0/-107 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:3.0:1788201411.492853:0:2362:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:3.0:1788201411.492856:0:2362:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:3.0:1788201411.492857:0:2362:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:3.0:1788201411.492861:0:2362:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201411.492865:0:2362:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 626 00000100:00080000:3.0:1788201411.492916:0:2362:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788201411.493155:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442757120:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201411.495247:0:2362:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442758784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201411.495271:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442758400:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201411.495287:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442757632:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201411.495297:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442756608:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201411.495828:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442758784:192.168.201.130@tcp:250:ptlrpcd_01_01.0 00000100:00080000:0.0:1788201411.495962:0:2358:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff90234598d880 x1875064442758656/t0(0) o9->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201421 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201411.495971:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:0.0:1788201411.495974:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442758656:192.168.201.130@tcp:9:kworker.0 00000100:00080000:0.0:1788201411.499689:0:2358:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.201.130@tcp: connect to target with instance 0 10000000:01000000:0.0:1788201411.499706:0:2358:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:0.0:1788201411.499713:0:2358:0:(import.c:1284:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x9383f985bc4d4c03 to 0x9383f985bc4d5cdb 00000100:00080000:0.0:1788201411.509947:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:0.0:1788201411.509965:0:2358:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.201.130@tcp; invalidating 00000100:00100000:2.0:1788201411.510424:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442758784:192.168.201.130@tcp:250:ptlrpcd_01_01.0 00000100:00080000:0.0:1788201411.510481:0:18057:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.201.130@tcp to MGS@192.168.201.130@tcp 00000100:00100000:0.0:1788201411.510485:0:18057:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:0.0:1788201411.510488:0:18057:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:0.0:1788201411.510504:0:18057:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff902345803600/0x4605c7c97e1c5231 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0x9383f985bc4d4c11 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201411.510516:0:18057:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff902345803600/0x4605c7c97e1c5231 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0x9383f985bc4d4c11 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201411.510522:0:18057:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201411.510536:0:18057:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201411.510546:0:18057:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff902345803600/0x4605c7c97e1c5231 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0x9383f985bc4d4c11 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201411.510558:0:18057:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff902342db1a00/0x4605c7c97e1c523f lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x9383f985bc4d4c1f expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201411.510565:0:18057:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff902342db1a00/0x4605c7c97e1c523f lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x9383f985bc4d4c1f expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201411.510571:0:18057:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201411.510574:0:18057:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201411.510576:0:18057:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff902342db1a00/0x4605c7c97e1c523f lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x9383f985bc4d4c1f expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201411.510588:0:18057:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff902345802000/0x4605c7c97e1c5262 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x9383f985bc4d4d22 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201411.510593:0:18057:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff902345802000/0x4605c7c97e1c5262 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x9383f985bc4d4d22 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201411.510598:0:18057:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201411.510602:0:18057:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201411.510604:0:18057:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff902345802000/0x4605c7c97e1c5262 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x9383f985bc4d4d22 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201411.510614:0:18057:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff902342db0c00/0x4605c7c97e1c5238 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x9383f985bc4d4c18 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201411.510621:0:18057:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff902342db0c00/0x4605c7c97e1c5238 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x9383f985bc4d4c18 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201411.510625:0:18057:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:0.0:1788201411.510628:0:18057:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201411.510631:0:18057:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff902342db0c00/0x4605c7c97e1c5238 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x9383f985bc4d4c18 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1788201411.510641:0:18057:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:0.0:1788201411.510645:0:18057:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from RECOVER to FULL 10000000:01000000:0.0:1788201411.510647:0:18057:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:0.0:1788201411.510650:0:18057:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:0.0:1788201411.510653:0:18057:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.201.130@tcp: Connection restored to 192.168.201.130@tcp (at 192.168.201.130@tcp) 00000100:00000400:3.0:1788201412.225024:0:2361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201396/real 1788201396] req@ffff90234598e300 x1875064442756736/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201412 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201412.259548:0:2361:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff90234598e300 x1875064442756736/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201412 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201412.262328:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442756736:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201412.269939:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788201413.121034:0:5601:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788201413.121043:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-cliir-ffff902350a11000 from 15 10000000:01000000:1.0:1788201413.121049:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788201413.121106:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff90235265ce00/0x4605c7c97e1c58ac lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201413.121119:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff90235265ce00/0x4605c7c97e1c58ac lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201413.121131:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff90235265ce00/0x4605c7c97e1c58ac lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201413.121159:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442758912:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201413.121245:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234363f540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201413.128757:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442758912:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201413.128788:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff90235265ce00/0x4605c7c97e1c58ac lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d5ce2 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201413.128801:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff90235265ce00/0x4605c7c97e1c58ac lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d5ce2 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201413.129615:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442759040:192.168.201.130@tcp:256:ll_cfg_requeue.0 00000100:00100000:1.0:1788201413.130203:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234363f540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201413.157035:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442759040:192.168.201.130@tcp:256:ll_cfg_requeue.0 10000000:01000000:1.0:1788201413.157084:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'lustre-cliir' failed (-2). 00010000:00010000:1.0:1788201413.157095:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff90235265ce00/0x4605c7c97e1c58ac lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d5ce2 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201413.157106:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff90235265ce00/0x4605c7c97e1c58ac lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d5ce2 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00000400:1.0:1788201413.157112:0:5601:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.130@tcp: IR log lustre-cliir failed, not fatal: rc = -2 10000000:01000000:1.0:1788201413.167479:0:5601:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:1.0:1788201413.167488:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-client-ffff902350a11000 from 42 10000000:01000000:1.0:1788201413.167495:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:1.0:1788201413.167558:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff90235265c000/0x4605c7c97e1c58b3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201413.168136:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff90235265c000/0x4605c7c97e1c58b3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201413.168150:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff90235265c000/0x4605c7c97e1c58b3 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201413.168184:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442759168:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201413.168283:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023413ef6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201413.171944:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442759168:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201413.171994:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff90235265c000/0x4605c7c97e1c58b3 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d5ce9 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201413.172007:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff90235265c000/0x4605c7c97e1c58b3 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d5ce9 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201413.172090:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442759296:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201413.172167:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023413ef6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201413.229519:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442759296:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201413.229599:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442759424:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788201413.229704:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023413ef6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201413.235188:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442759424:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000040:00080000:2.0:1788201413.240634:0:18059:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 42 00000020:01000000:1.0:1788201413.241241:0:5601:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 42-41 (rc=0) 10000000:01000000:1.0:1788201413.241261:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:1.0:1788201413.241272:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff90235265c000/0x4605c7c97e1c58b3 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d5ce9 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201413.241285:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff90235265c000/0x4605c7c97e1c58b3 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d5ce9 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201413.241296:0:5601:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:1.0:1788201413.241299:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log params-ffff902350a11000 from 7 10000000:01000000:1.0:1788201413.241302:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:1.0:1788201413.241342:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff90235265d600/0x4605c7c97e1c58ba lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201413.241352:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff90235265d600/0x4605c7c97e1c58ba lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201413.241364:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff90235265d600/0x4605c7c97e1c58ba lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201413.241395:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442759552:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201413.241489:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234363f540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201413.250953:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442759552:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201413.250995:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff90235265d600/0x4605c7c97e1c58ba lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d5cf0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201413.251009:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff90235265d600/0x4605c7c97e1c58ba lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d5cf0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201413.251084:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442759680:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201413.251166:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234363f540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201413.259132:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442759680:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201413.259195:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442759808:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788201413.259304:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234363f540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201413.272688:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442759808:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000040:00080000:2.0:1788201413.273871:0:18060:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000020:01000000:1.0:1788201413.274157:0:5601:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:1.0:1788201413.274175:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:1.0:1788201413.274187:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff90235265d600/0x4605c7c97e1c58ba lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d5cf0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201413.274197:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff90235265d600/0x4605c7c97e1c58ba lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d5cf0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201413.274212:0:5601:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:1.0:1788201413.274215:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-sptlrpc-ffff902350bf19c0 from 1 10000000:01000000:1.0:1788201413.274218:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:1.0:1788201413.274254:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff90235265c400/0x4605c7c97e1c58c1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201413.274262:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff90235265c400/0x4605c7c97e1c58c1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201413.274273:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff90235265c400/0x4605c7c97e1c58c1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201413.274297:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442759936:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201413.279722:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023413ef6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201413.282032:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442759936:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201413.282061:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff90235265c400/0x4605c7c97e1c58c1 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d5cf7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201413.282074:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff90235265c400/0x4605c7c97e1c58c1 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d5cf7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201413.282155:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442760064:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201413.282225:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023413ef6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201413.296137:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442760064:192.168.201.130@tcp:501:ll_cfg_requeue.0 10000000:01000000:1.0:1788201413.296186:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:1.0:1788201413.296198:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff90235265c400/0x4605c7c97e1c58c1 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d5cf7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201413.296210:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff90235265c400/0x4605c7c97e1c58c1 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d5cf7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201413.313260:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201414.337399:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201415.362288:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201415.492644:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201415.492651:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788201415.492770:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442760192:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201415.492977:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201415.492984:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442760320:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201415.498406:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442760192:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201415.499988:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442760320:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201416.385163:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201416.513093:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201416.513102:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788201416.513221:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201416.513245:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442760448:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201416.514523:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442760576:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201416.523046:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442760448:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201416.532550:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442760576:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201417.409106:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1788201417.537258:0:2360:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201401/real 1788201401] req@ffff9023450b8000 x1875064442757248/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201417 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201417.537294:0:2360:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9023450b8000 x1875064442757248/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201417 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201417.537316:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442757248:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201418.433556:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201419.457122:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201420.162474:0:17860:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c58c8 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201420.162495:0:17860:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c58c8 lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 3 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00080000:1.0:1788201420.162596:0:2359:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff90234519ce00 x1875064442760704/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 664/4320 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1788201420.481099:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201420.674148:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201410/real 1788201410] req@ffff90234519d880 x1875064442758272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'ptlrpcd_00_01.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201420.674179:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442758272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201420 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'ptlrpcd_00_01.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201420.674194:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201420.678676:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201420.678687:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442758272:192.168.201.130@tcp:38:ptlrpcd_00_01.0 00000100:00100000:0.0:1788201420.678971:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442760832:192.168.201.130@tcp:400:kworker.0 00000100:00080000:1.0:1788201420.679607:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201420.679613:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201420.679617:0:37:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201420.679624:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 687 00000100:00080000:1.0:1788201420.679632:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1788201420.679639:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201420.679649:0:37:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201420.679673:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442760960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201420.679739:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201420.679747:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442761088:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201420.679851:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442760960:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201420.683208:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442761088:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201420.697100:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:1.0:1788201420.697326:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442760832:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201420.714586:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458276352 00000100:00100000:2.0:1788201420.714607:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458276352 00000100:00100000:2.0:1788201420.714708:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1361 00000100:00100000:2.0:1788201420.714720:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90235268bb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:7ad49433-54fa-4e21-b756-859420d95603+3:18167:x1875064458276352:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201420.714755:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.130@tcp lock: ffff90235265ce00/0x4605c7c97e1c58ac lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x9383f985bc4d5ce2 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201420.714893:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201420.714901:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90235268bb80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:7ad49433-54fa-4e21-b756-859420d95603+3:18167:x1875064458276352:12345-192.168.201.130@tcp:104: Request processed in 183us (17816us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201420.714910:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1361 00010000:00010000:0.0:1788201420.715069:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201420.715084:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.130@tcp lock: ffff90235265ce00/0x4605c7c97e1c58ac lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x9383f985bc4d5ce2 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201420.715099:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265ce00 already unused, calling callback (ffffffffc12bbc20) 10000000:00010000:0.0:1788201420.715104:0:5599:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.130@tcp lock: ffff90235265ce00/0x4605c7c97e1c58ac lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x9383f985bc4d5ce2 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201420.715114:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff90235265ce00/0x4605c7c97e1c58ac lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x9383f985bc4d5ce2 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201420.715120:0:5599:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff90235265ce00/0x4605c7c97e1c58ac lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x9383f985bc4d5ce2 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201420.715126:0:5599:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201420.715184:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.130@tcp lock: ffff90235265ce00/0x4605c7c97e1c58ac lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x9383f985bc4d5ce2 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201420.715191:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201420.715218:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.130@tcp lock: ffff90235265ce00/0x4605c7c97e1c58ac lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x9383f985bc4d5ce2 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201420.715225:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff90235265ce00/0x4605c7c97e1c58ac lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x9383f985bc4d5ce2 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201420.715246:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201420.715251:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201420.715253:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201420.715270:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201420.715303:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442761216:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00080000:0.0:1788201420.721137:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201420.721152:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-19) 00000100:00100000:0.0:1788201420.721158:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442760960:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788201420.722505:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442761216:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201421.098861:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201421.099109:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458280576 00000100:00100000:3.0:1788201421.099121:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458280576 00000100:00100000:3.0:1788201421.099226:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1362 00000100:00100000:3.0:1788201421.099238:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234598e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6703:x1875064458280576:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201421.099258:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201421.099347:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201421.099354:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234598e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:6703:x1875064458280576:12345-192.168.201.130@tcp:104: Request processed in 119us (503us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201421.099361:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1362 00010000:00010000:1.0:1788201421.099426:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201421.099435:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201421.099450:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865b000 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:1.0:1788201421.099456:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201421.099540:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201421.099548:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201421.099579:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201421.099587:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e8e6f774/0x4605c7c97e1c5548 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x46e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58d6 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201421.099610:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201421.099612:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201421.099613:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201421.099840:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442761344:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201421.120872:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442761344:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201421.129693:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201421.130058:0:5590:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458282368 00000100:00100000:2.0:1788201421.130068:0:5590:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458282368 00000100:00100000:2.0:1788201421.130116:0:5590:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1363 00000100:00100000:2.0:1788201421.130124:0:5590:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902352688700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13442:x1875064458282368:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201421.130143:0:5590:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201421.130223:0:5590:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201421.130229:0:5590:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902352688700 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:LOV_OSC_UUID+4:13442:x1875064458282368:12345-192.168.201.130@tcp:104: Request processed in 106us (547us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201421.130235:0:5590:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1363 00010000:00010000:1.0:1788201421.130272:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201421.130280:0:5598:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201421.130290:0:5598:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865a600 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:1.0:1788201421.130296:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201421.130399:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201421.130408:0:5598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201421.130445:0:5598:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201421.130454:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000562f63a3/0x4605c7c97e1c5564 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x470:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58f9 expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201421.130480:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201421.130483:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201421.130485:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201421.130502:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201421.130516:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442761472:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201421.131945:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201421.132120:0:5591:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1788201421.132137:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458282880 00000100:00100000:2.0:1788201421.132143:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458282880 00000100:00100000:2.0:1788201421.132189:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1364 00000100:00100000:2.0:1788201421.132205:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff9023584a4c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13446:x1875064458282880:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201421.132219:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5bc3 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201421.132407:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201421.132414:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff9023584a4c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13446:x1875064458282880:12345-192.168.201.130@tcp:104: Request processed in 210us (477us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201421.132420:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1364 00010000:00010000:1.0:1788201421.133851:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201421.133862:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5bc3 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201421.133888:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265d400 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:1.0:1788201421.133896:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d5bc3 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201421.133990:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5bc3 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201421.134005:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201421.134032:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5bc3 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201421.134041:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b9ec767d/0x4605c7c97e1c5897 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x466:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5bc3 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201421.134063:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201421.134065:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201421.134067:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201421.134087:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442761600:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201421.142569:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201421.142600:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201421.142622:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201421.142644:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458283008 00000100:00100000:2.0:1788201421.142654:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458283008 00000100:00100000:2.0:1788201421.142689:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458283136 00000100:00100000:2.0:1788201421.142691:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458283136 00000100:00100000:2.0:1788201421.142700:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458283264 00000100:00100000:2.0:1788201421.142702:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458283264 00000100:00100000:2.0:1788201421.142713:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1365 00000100:00100000:2.0:1788201421.142735:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234598e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+6:10484:x1875064458283008:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201421.142755:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5b37 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201421.142842:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201421.142849:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234598e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+6:10484:x1875064458283008:12345-192.168.201.130@tcp:104: Request processed in 129us (290us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201421.142857:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1365 00000100:00100000:2.0:1788201421.142896:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1366 00010000:00010000:0.0:1788201421.142908:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00000100:00100000:2.0:1788201421.142908:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234598c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13441:x1875064458283136:12345-192.168.201.130@tcp:104: 00010000:00010000:0.0:1788201421.142920:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5b37 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201421.142921:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201421.142944:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265c800 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:0.0:1788201421.142952:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d5b37 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201421.142968:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201421.142974:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234598c380 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+5:13441:x1875064458283136:12345-192.168.201.130@tcp:104: Request processed in 71us (374us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201421.142979:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1366 00000100:00100000:2.0:1788201421.142997:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1367 00010000:00010000:0.0:1788201421.143042:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 2/0,0 mode: --/PR res: [0x240000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5b37 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201421.143051:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201421.143086:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 1/0,0 mode: --/PR res: [0x240000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5b37 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201421.143095:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058b6731d/0x4605c7c97e1c587b lrc: 0/0,0 mode: --/PR res: [0x240000400:0x464:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5b37 expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201421.143124:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201421.143128:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201421.143135:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865ba00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:0.0:1788201421.143138:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201421.143173:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201421.143182:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201421.143189:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201421.143197:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007b0d08f9/0x4605c7c97e1c5556 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x46f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d58dd expref: -99 pid: 17860 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201421.143215:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201421.143218:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201421.143220:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201421.143471:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442761728:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201421.143547:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201421.143554:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442761856:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201421.143584:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201421.144513:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234598d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13448:x1875064458283264:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201421.144529:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5b7d expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201421.144600:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201421.144607:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234598d180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:13448:x1875064458283264:12345-192.168.201.130@tcp:104: Request processed in 1604us (1983us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201421.144616:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1367 00010000:00010000:1.0:1788201421.145264:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201421.145436:0:5598:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d5b7d expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201421.145448:0:5598:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265ca00 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:1.0:1788201421.145454:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d5b7d expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201421.145591:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5b7d expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201421.145600:0:5598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201421.145625:0:5598:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5b7d expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201421.145634:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011e1d9e0/0x4605c7c97e1c5889 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x465:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d5b7d expref: -99 pid: 17859 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201421.145654:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201421.145656:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201421.145658:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201421.145799:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442761984:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201421.146358:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442761472:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201421.147665:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442761600:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201421.153751:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442761856:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201421.153767:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442761728:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201421.154832:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442761984:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201421.507249:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788201421.900607:0:5601:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788201421.900615:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-cliir-ffff902350a11000 from 15 10000000:01000000:2.0:1788201421.900622:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788201421.900669:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff902345107800/0x4605c7c97e1c58cf lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201421.900683:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff902345107800/0x4605c7c97e1c58cf lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201421.900695:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff902345107800/0x4605c7c97e1c58cf lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201421.900832:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442762112:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201421.900957:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201421.903204:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442762112:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201421.903235:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff902345107800/0x4605c7c97e1c58cf lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d5e24 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201421.903259:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff902345107800/0x4605c7c97e1c58cf lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d5e24 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201421.903339:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442762240:192.168.201.130@tcp:256:ll_cfg_requeue.0 00000100:00100000:1.0:1788201421.903418:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351d29cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201421.926353:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442762240:192.168.201.130@tcp:256:ll_cfg_requeue.0 00000100:00080000:2.0:1788201421.933245:0:5601:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 12, was 0): DISCONN 00010000:00080000:2.0:1788201421.933262:0:5601:0:(ldlm_lib.c:85:import_set_conn()) imp ffff902348897000@lustre-MDT0000-mdc-ffff902350a11000: found existing conn 192.168.201.130@tcp, moved to head 00000100:00080000:2.0:1788201421.933270:0:5601:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201421.933276:0:5601:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 0 00000100:00080000:2.0:1788201421.933285:0:5601:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788201421.933297:0:5601:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201421.933331:0:5601:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ea00 x1875064442762368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 10000000:01000000:2.0:1788201421.933396:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:2.0:1788201421.933404:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff902345107800/0x4605c7c97e1c58cf lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d5e24 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201421.933413:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff902345107800/0x4605c7c97e1c58cf lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d5e24 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201421.933511:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442762368:192.168.201.130@tcp:38:ll_cfg_requeue.0 00010000:00080000:1.0:1788201421.943691:0:2358:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff90234598ea00 x1875064442762368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/416 e 0 to 0 dl 1788201431 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201421.943726:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 17179869289 gen 2 00000100:00080000:1.0:1788201421.944005:0:2358:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff902350a11000: connect to target with instance 5 00000100:00080000:1.0:1788201421.944024:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:1.0:1788201421.944029:0:2358:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788201421.944032:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 17179869289 gen 2 00000100:00080000:1.0:1788201421.944035:0:2358:0:(recover.c:53:ptlrpc_replay_next()) import ffff902348897000 from lustre-MDT0000_UUID committed 17179869289 last 0 00000100:00080000:1.0:1788201421.944038:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:1.0:1788201421.945021:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442762368:192.168.201.130@tcp:38:ll_cfg_requeue.0 00010000:00080000:2.0:1788201421.945100:0:18071:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff902350a11000 to lustre-MDT0000_UUID@192.168.201.130@tcp 00010000:00010000:2.0:1788201421.945106:0:18071:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff902350a11000 (2) 00010000:00010000:2.0:1788201421.945152:0:18071:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685e00/0x4605c7c97e1c536c lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x9383f985bc4d5334 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201421.945167:0:18071:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff902350a11000 00010000:00010000:2.0:1788201421.945653:0:18071:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5318 expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00010000:00080000:2.0:1788201421.945680:0:18071:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00000100:00100000:1.0:1788201421.945950:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442762496:192.168.201.130@tcp:101:ldlm_lock_repla.0 00010000:00010000:1.0:1788201421.950018:0:2358:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00000100:00080000:1.0:1788201421.950037:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:1.0:1788201421.950071:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442762496:192.168.201.130@tcp:101:ldlm_lock_repla.0 00000100:00100000:1.0:1788201421.950098:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442762624:192.168.201.130@tcp:400:ptlrpcd_rcv.0 00000100:00080000:1.0:1788201422.018252:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:1.0:1788201422.018274:0:2358:0:(client.c:3234:ptlrpc_resend_req()) @@@ going to resend req@ffff90234519c380 x1875064442762624/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201437 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201422.018287:0:2358:0:(client.c:3243:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff90234519c380 x1875064442762624/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201437 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201422.018293:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:1.0:1788201422.019089:0:2358:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff902350a11000: Connection restored to 192.168.201.130@tcp (at 192.168.201.130@tcp) 00000100:00080000:1.0:1788201422.019099:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000041905aa6): req@ffff90234519ce00 x1875064442760704/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 664/4320 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201422.019124:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000002a5d3e6): req@ffff9023450b8a80 x1875064442758144/t0(0) o63->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:24/4 lens 664/224 e 0 to 0 dl 1788201426 ref 1 fl Rpc:RSQU/200/0 rc -107/-107 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201422.019139:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442762624:192.168.201.130@tcp:400:ptlrpcd_rcv.0 00000100:00080000:1.0:1788201422.019177:0:2360:0:(client.c:3727:ptlrpc_set_mbits()) resend with new mbits old x1875064442758144 new x1875064442762752 00010000:00010000:1.0:1788201422.075468:0:2360:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c58a5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x0 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201422.075497:0:2360:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c58a5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201422.075504:0:2360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c58a5 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201422.075511:0:2360:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c58a5 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201422.075520:0:2360:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c58a5 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201422.075528:0:2360:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865a200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201422.075532:0:2360:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c58a5 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201422.075540:0:2360:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c58a5 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x206400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201422.075546:0:2360:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c58a5 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x20e400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201422.075552:0:2360:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201422.075558:0:2360:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c58a5 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201422.075565:0:2360:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a200/0x4605c7c97e1c58a5 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x402:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 17241 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201422.075618:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442758144:192.168.201.130@tcp:63:ll_sa.0 00010000:00010000:0.0:1788201422.080543:0:2359:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c58c8 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x0 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201422.080565:0:2359:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c58c8 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201422.080573:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c58c8 lrc: 4/1,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201422.080580:0:2359:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c58c8 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201422.080589:0:2359:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c58c8 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201422.080596:0:2359:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265d000 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201422.080601:0:2359:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c58c8 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201422.080608:0:2359:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c58c8 lrc: 4/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x206400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201422.080620:0:2359:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c58c8 lrc: 5/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x20e400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201422.080627:0:2359:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201422.080633:0:2359:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c58c8 lrc: 3/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201422.080640:0:2359:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d000/0x4605c7c97e1c58c8 lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x40a:0x0].0x0 bits 0x3/0x3 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 17860 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201422.080677:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:0:1875064442760704:192.168.201.130@tcp:63:ll_sa.0 00000080:00200000:1.0:1788201422.153041:0:17241:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x407:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201422.153057:0:17241:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201422.153118:0:17241:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x407:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201422.153135:0:17241:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x407:0x0](ffff9023449b4b08) 00000080:00200000:1.0:1788201422.153267:0:17241:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x408:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201422.153290:0:17241:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201422.153312:0:17241:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x408:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201422.153322:0:17241:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x408:0x0](ffff9023449ab248) 00000080:00200000:1.0:1788201422.153372:0:17241:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x409:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201422.153376:0:17241:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201422.153395:0:17241:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x409:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201422.153404:0:17241:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x409:0x0](ffff9023449adb88) 00000080:00200000:1.0:1788201422.153457:0:17241:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x402:0x0](ffff9023449a1988) 00010000:00010000:0.0:1788201422.529352:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1788201422.721233:0:2362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201406/real 1788201406] req@ffff90234598c700 x1875064442757760/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201422 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201422.721278:0:2362:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff90234598c700 x1875064442757760/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201422 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201422.721298:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442757760:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201423.553242:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201424.577915:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201425.601246:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201425.793266:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442762880:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201425.793371:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442763136:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201425.793404:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201425.793409:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442763008:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201425.801223:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442763008:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201425.801242:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442762880:192.168.201.130@tcp:400:kworker.0 00010000:00010000:1.0:1788201425.801274:0:2359:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00100000:1.0:1788201425.801280:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442763136:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201425.801415:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201425.801439:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201425.801441:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201425.801443:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201426.625197:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201427.661576:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201428.673428:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201429.697535:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201430.734041:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201430.913109:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442763264:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201430.913209:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442763520:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201430.913243:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201430.913249:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442763392:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201430.926501:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442763392:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201430.926518:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442763264:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201430.926528:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442763520:192.168.201.130@tcp:400:kworker.0 00000080:00200008:2.0:1788201431.510790:0:18160:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201431.510806:0:18160:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201431.510829:0:18160:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201431.510848:0:18160:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00010000:00010000:2.0:1788201431.510873:0:18160:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00000080:00200000:2.0:1788201431.510902:0:18160:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201431.510911:0:18160:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201431.510915:0:18160:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201431.510924:0:18160:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b180ddbcb8 00010000:00010000:2.0:1788201431.510928:0:18160:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00010000:00010000:2.0:1788201431.510937:0:18160:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00000080:00200008:2.0:1788201431.511520:0:18160:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788201431.511526:0:18160:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201431.511535:0:18160:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201431.511560:0:18160:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00010000:00010000:2.0:1788201431.511571:0:18160:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00000080:00200000:2.0:1788201431.511578:0:18160:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201431.511583:0:18160:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201431.511586:0:18160:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201431.511591:0:18160:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b180ddbcb8 00010000:00010000:2.0:1788201431.511594:0:18160:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00010000:00010000:2.0:1788201431.511607:0:18160:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00000080:00200000:2.0:1788201431.511701:0:18160:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201431.511709:0:18160:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201431.511716:0:18160:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00010000:00010000:2.0:1788201431.511726:0:18160:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00000080:00200000:2.0:1788201431.511733:0:18160:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201431.511738:0:18160:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201431.511740:0:18160:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:2.0:1788201431.511744:0:18160:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b180ddbbb0 00010000:00010000:2.0:1788201431.511748:0:18160:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00010000:00010000:2.0:1788201431.511756:0:18160:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00000080:00200000:2.0:1788201431.511765:0:18160:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 44 00000080:00200008:2.0:1788201431.511772:0:18160:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00000080:00200080:2.0:1788201431.511853:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:2.0:1788201431.511867:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192c00 uarg=0000000000000000 00800000:00000004:2.0:1788201431.511873:0:18160:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: age 657, max_age 707 00000100:00100000:2.0:1788201431.512067:0:18160:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:18160:1875064442763648:192.168.201.130@tcp:41:lfs.0 00000100:00100000:2.0:1788201431.512164:0:18160:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342fe3480 going to sleep for 11 seconds 00000100:00100000:0.0:1788201431.514579:0:18160:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:18160:1875064442763648:192.168.201.130@tcp:41:lfs.0 00800000:00000004:0.0:1788201431.514613:0:18160:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: update ffff902350be5190 cache blocks 17239/17271 objects 530726/531060 00000080:00200080:0.0:1788201431.514697:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514716:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514723:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514728:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514732:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514737:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514741:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514745:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514748:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514751:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514755:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514759:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514762:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514766:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514770:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514773:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514777:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514781:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514785:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514789:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514793:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514796:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514800:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514804:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514807:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514811:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514815:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514818:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514822:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514826:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514830:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514833:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514837:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514841:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514844:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514848:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514852:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514855:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514859:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514863:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514867:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514870:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514904:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514910:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514915:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514918:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514921:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514924:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514928:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514931:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514935:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514938:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514941:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514944:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514948:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514951:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514954:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514958:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514962:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514965:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514969:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514972:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514976:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514979:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514983:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514986:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514989:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.514993:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.514996:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515000:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515003:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515006:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515010:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515013:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515016:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515019:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515022:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515025:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515029:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515032:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515037:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515040:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515044:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515047:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515050:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515054:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515058:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515061:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515065:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515068:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515072:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515075:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515079:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515083:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515087:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515090:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515094:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515097:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515100:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515103:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515107:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515110:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515113:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515116:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515119:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515123:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515127:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515131:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515134:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515138:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515142:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515146:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515150:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515153:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515156:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515160:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515164:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515168:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515171:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515174:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515179:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515182:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515185:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515189:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515192:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515195:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515199:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00800000:00000080:0.0:1788201431.515202:0:18160:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.515206:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.515211:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00020000:00000004:0.0:1788201431.515219:0:18160:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: age 657, max_age 707 00000100:00100000:0.0:1788201431.515256:0:18160:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:18160:1875064442763776:192.168.201.130@tcp:13:lfs.0 00000100:00100000:0.0:1788201431.515333:0:18160:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436c0780 going to sleep for 11 seconds 00000100:00100000:0.0:1788201431.523998:0:18160:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:18160:1875064442763776:192.168.201.130@tcp:13:lfs.0 00020000:00000004:0.0:1788201431.524026:0:18160:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: update ffff902350be1e10 cache blocks 3667/3683 objects 117728/118665 00000080:00200080:0.0:1788201431.524092:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.524107:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00020000:00000004:0.0:1788201431.524115:0:18160:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: age 657, max_age 707 00000100:00080000:0.0:1788201431.524125:0:18160:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 345 00000100:00080000:0.0:1788201431.524135:0:18160:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201431.524144:0:18160:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 488505344, lost: 0. 00000100:00100000:0.0:1788201431.524165:0:18160:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442763904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201431.524207:0:18160:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9023450b8a80 x1875064442764032/t0(0) o13->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201431.524216:0:18160:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436c0780 going to sleep for 0 seconds 00000100:00100000:2.0:1788201431.524821:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442763904:192.168.201.130@tcp:8:lfs.0 00000100:00100000:2.0:1788201431.530115:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff902350a11000: committing for last_committed 4294968873 gen 3 00000100:00080000:2.0:1788201431.530136:0:2358:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff902350a11000: connect to target with instance 1 00000100:00080000:2.0:1788201431.530147:0:2358:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff902350a11000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788201431.530161:0:2358:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:2.0:1788201431.530165:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:2.0:1788201431.530175:0:2358:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788201431.530182:0:2358:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:2.0:1788201431.530200:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000009948998a): req@ffff9023450b8a80 x1875064442764032/t0(0) o13->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201431.530216:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442763904:192.168.201.130@tcp:8:lfs.0 00000100:00100000:0.0:1788201431.530381:0:18160:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436c0780 going to sleep for 11 seconds 00000100:00100000:0.0:1788201431.533688:0:18160:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:18160:1875064442764032:192.168.201.130@tcp:13:lfs.0 00020000:00000004:0.0:1788201431.533719:0:18160:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: update ffff902351d65190 cache blocks 3679/3683 objects 117792/118185 00000080:00200080:0.0:1788201431.533801:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533821:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533829:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533834:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533838:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533843:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533846:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533850:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533854:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533858:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533861:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533865:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533868:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533871:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533915:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533922:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533927:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533930:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533934:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533938:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533941:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533945:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533949:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533953:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533956:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533960:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533964:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533967:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533971:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533974:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533978:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533981:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533984:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533987:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533990:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533993:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.533996:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.533999:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534002:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534005:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534008:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534011:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534015:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534018:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534021:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534024:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534028:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534031:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534035:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534038:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534042:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534045:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534049:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534052:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534056:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534060:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534064:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534068:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534072:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534075:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534079:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534082:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534085:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534088:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534091:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534094:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534098:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534100:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534103:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534106:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534109:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534112:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534115:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534118:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534121:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534124:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534126:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534129:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534132:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534135:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534138:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534140:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534143:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534146:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534149:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534152:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534155:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534158:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534162:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534164:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534168:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534171:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534175:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534178:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534182:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534185:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534189:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534192:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534196:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534198:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534201:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534204:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534207:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534210:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534213:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534216:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534220:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534222:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534225:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534228:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534231:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534233:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534237:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534239:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534242:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534245:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534248:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534251:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534254:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534256:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534259:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534261:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534265:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534267:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200080:0.0:1788201431.534270:0:18160:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fffbcf6ea00 00020000:00000080:0.0:1788201431.534273:0:18160:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587e400 uarg=0000000000000000 00000080:00200008:0.0:1788201431.534287:0:18160:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00010000:00010000:0.0:1788201431.747762:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201432.769073:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201433.795023:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201434.818034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201435.841244:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201436.033174:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442764160:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201436.033301:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442764416:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201436.033336:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788201436.033343:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442764288:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201436.033369:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442764544:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201436.035813:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442764288:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201436.035832:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442764544:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201436.035841:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442764160:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201436.035851:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442764416:192.168.201.130@tcp:400:kworker.0 00010000:00010000:1.0:1788201436.867100:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201437.889172:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201438.119445:0:18439:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 00010000:00010000:1.0:1788201438.914043:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201439.817706:0:18585:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:2.0:1788201439.937822:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201440.962079:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201441.153219:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442764800:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201441.153340:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442765056:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201441.153406:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788201441.153414:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442764672:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201441.153445:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442764928:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201441.159721:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442764672:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201441.169094:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442764928:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201441.169117:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442764800:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201441.169128:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442765056:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201441.985507:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788201442.326348:0:18624:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201442.329042:0:18624:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201442.329079:0:18624:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201442.329102:0:18624:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00010000:00010000:3.0:1788201442.329120:0:18624:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00000080:00200000:3.0:1788201442.329128:0:18624:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201442.329139:0:18624:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201442.329142:0:18624:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:3.0:1788201442.329151:0:18624:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b180c87cb8 00010000:00010000:3.0:1788201442.329155:0:18624:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00010000:00010000:3.0:1788201442.329167:0:18624:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00000080:00200008:3.0:1788201442.330463:0:18624:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788201442.330474:0:18624:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201442.330492:0:18624:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201442.330507:0:18624:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00010000:00010000:3.0:1788201442.330521:0:18624:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00000080:00200000:3.0:1788201442.330530:0:18624:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201442.330538:0:18624:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201442.330542:0:18624:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:3.0:1788201442.330549:0:18624:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b180c87cb8 00010000:00010000:3.0:1788201442.330553:0:18624:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00010000:00010000:3.0:1788201442.330561:0:18624:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00000080:00200000:3.0:1788201442.330793:0:18624:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201442.330801:0:18624:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201442.330809:0:18624:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00010000:00010000:3.0:1788201442.330823:0:18624:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00000080:00200000:3.0:1788201442.330829:0:18624:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201442.330833:0:18624:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201442.330835:0:18624:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:3.0:1788201442.330840:0:18624:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b180c87bb0 00010000:00010000:3.0:1788201442.330842:0:18624:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00010000:00010000:3.0:1788201442.330850:0:18624:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 97 lru_type: 1 00000080:00200000:3.0:1788201442.330858:0:18624:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 44 00000080:00200008:3.0:1788201442.330866:0:18624:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00000080:00200080:3.0:1788201442.331241:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.331264:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5000 uarg=0000000000000000 00800000:00000004:3.0:1788201442.331273:0:18624:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: age 708, max_age 717 00000100:00100000:3.0:1788201442.331341:0:18624:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:18624:1875064442765184:192.168.201.130@tcp:41:lfs.0 00000100:00100000:3.0:1788201442.331504:0:18624:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005a780 going to sleep for 11 seconds 00000100:00100000:3.0:1788201442.335280:0:18624:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:18624:1875064442765184:192.168.201.130@tcp:41:lfs.0 00800000:00000004:3.0:1788201442.335321:0:18624:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: update ffff902350be5190 cache blocks 17239/17271 objects 530726/531060 00000080:00200080:3.0:1788201442.335425:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.335447:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5000 uarg=0000000000000000 00000080:00200080:3.0:1788201442.335457:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.335463:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5000 uarg=0000000000000000 00000080:00200080:3.0:1788201442.335468:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.335472:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5000 uarg=0000000000000000 00000080:00200080:3.0:1788201442.335476:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.335479:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5000 uarg=0000000000000000 00000080:00200080:3.0:1788201442.335483:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.335487:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5000 uarg=0000000000000000 00000080:00200080:3.0:1788201442.335490:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.335493:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5000 uarg=0000000000000000 00000080:00200080:3.0:1788201442.335498:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.335501:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5000 uarg=0000000000000000 00000080:00200080:3.0:1788201442.335505:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.335508:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5000 uarg=0000000000000000 00000080:00200080:3.0:1788201442.335526:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.335532:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5000 uarg=0000000000000000 00000080:00200080:3.0:1788201442.335536:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.335540:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5000 uarg=0000000000000000 00000080:00200080:3.0:1788201442.335544:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.335547:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5000 uarg=0000000000000000 00000080:00200080:3.0:1788201442.335552:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.335556:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5000 uarg=0000000000000000 00000080:00200080:3.0:1788201442.335560:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.335564:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5000 uarg=0000000000000000 00000080:00200080:3.0:1788201442.335569:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.335573:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5000 uarg=0000000000000000 00000080:00200080:3.0:1788201442.335577:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.336860:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.336896:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.336903:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.336907:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.336910:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.336914:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.336928:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.336933:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.336937:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.336941:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.336944:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.336948:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.336951:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.336955:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.336958:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.336963:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.336966:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.336970:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.336973:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.336977:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.336980:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.336983:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.336987:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.336991:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.336995:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.336998:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.337002:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.337005:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.337010:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.337013:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.337017:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.337020:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.337023:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.337028:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.337032:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.337035:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.337038:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.337042:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.337045:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.337049:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.337052:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.337056:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.337059:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.337062:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.337065:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.337069:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.337074:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.337078:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.337081:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.337086:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.337089:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.337093:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.337096:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.337100:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.337103:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.337106:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.337109:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.337113:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.337116:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.337119:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338064:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338084:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338089:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338095:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338099:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338104:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338108:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338113:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338117:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338660:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338782:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338793:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338799:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338804:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338809:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338812:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338816:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338819:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338822:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338826:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338829:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338833:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338835:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338839:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338842:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338846:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338849:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338853:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338856:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338860:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338863:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338867:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338871:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338904:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338910:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338926:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338931:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338935:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00800000:00000080:3.0:1788201442.338938:0:18624:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201442.338943:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:3.0:1788201442.338950:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00020000:00000004:3.0:1788201442.338960:0:18624:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: age 708, max_age 717 00000100:00100000:3.0:1788201442.339025:0:18624:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:18624:1875064442765312:192.168.201.130@tcp:13:lfs.0 00000100:00100000:3.0:1788201442.339272:0:18624:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005a780 going to sleep for 11 seconds 00000100:00100000:3.0:1788201442.361478:0:18624:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:18624:1875064442765312:192.168.201.130@tcp:13:lfs.0 00020000:00000004:3.0:1788201442.361532:0:18624:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: update ffff902350be1e10 cache blocks 3667/3683 objects 117728/118665 00000080:00200080:3.0:1788201442.361637:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:3.0:1788201442.361660:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00020000:00000004:3.0:1788201442.361683:0:18624:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: age 708, max_age 717 00000100:00100000:3.0:1788201442.361728:0:18624:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:18624:1875064442765440:192.168.201.130@tcp:13:lfs.0 00000100:00100000:3.0:1788201442.361820:0:18624:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005a780 going to sleep for 11 seconds 00000100:00100000:3.0:1788201442.366815:0:18624:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:18624:1875064442765440:192.168.201.130@tcp:13:lfs.0 00020000:00000004:3.0:1788201442.366861:0:18624:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: update ffff902351d65190 cache blocks 3679/3683 objects 117792/118185 00000080:00200080:0.0:1788201442.367175:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367277:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367287:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367293:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367298:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367302:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367306:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367310:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367315:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367319:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367323:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367327:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367331:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367335:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367339:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367343:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367347:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367363:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367368:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367372:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367376:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367379:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367383:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367387:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367390:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367394:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367398:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367402:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367406:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367410:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367414:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367418:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367422:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367425:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367429:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367432:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367436:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367439:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367443:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367447:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367451:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367454:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367458:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367462:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367466:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367470:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367474:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367477:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367481:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367484:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367489:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367492:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367496:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367499:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367503:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367506:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367510:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367514:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367518:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367521:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367525:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367528:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367531:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367535:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367539:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367542:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367546:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367549:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367554:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367558:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367562:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367566:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367570:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367574:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367578:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367582:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367586:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367589:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367593:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367597:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367601:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367605:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367610:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367614:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367617:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367621:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367626:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367629:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367633:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367637:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367641:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367644:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367648:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367652:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367656:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367660:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367664:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367785:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367792:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367796:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367800:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367803:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367807:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367811:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367815:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367819:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367824:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367827:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367831:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367835:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367839:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367843:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367847:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367850:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367854:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367858:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367862:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367865:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367869:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367872:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367918:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367923:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367927:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367931:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200080:0.0:1788201442.367935:0:18624:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffc1df1b6d0 00020000:00000080:0.0:1788201442.367940:0:18624:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587ec00 uarg=0000000000000000 00000080:00200008:0.0:1788201442.367956:0:18624:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00010000:00010000:0.0:1788201443.010148:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201444.035084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201445.059550:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201446.083102:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201446.276197:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442765568:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201446.276339:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442765824:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201446.276373:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788201446.276380:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442765696:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201446.276405:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442765952:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201446.283248:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442765696:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201446.283289:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442765952:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201446.283300:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442765568:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201446.283311:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442765824:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201447.107432:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201448.137847:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201449.158243:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201450.183460:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201450.609804:0:19205:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 110a: DNE: create striped dir, fail MDT1 ========================================================== 14:37:28 (1788201448) 00010000:00010000:0.0:1788201451.204919:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201451.394593:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442766080:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201451.394696:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442766336:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201451.394730:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788201451.394738:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442766208:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201451.394762:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442766464:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201451.410447:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442766464:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201451.410468:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442766336:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201451.414319:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442766208:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201451.414598:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442766080:192.168.201.130@tcp:400:kworker.0 00000001:02000400:1.0:1788201451.991463:0:19341:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_110a needs >= 2 MDTs 00010000:00010000:3.0:1788201452.236718:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201453.253124:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201453.887690:0:19494:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 110b: DNE: create striped dir, fail MDT1 and client ========================================================== 14:37:32 (1788201452) 00010000:00010000:0.0:1788201454.284928:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201455.298120:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201455.615188:0:19632:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_110b needs >= 2 MDTs 00010000:00010000:3.0:1788201456.321198:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201456.515699:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442766592:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201456.515802:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442766848:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201456.515841:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788201456.515847:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442766720:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201456.515873:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442766976:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201456.527554:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442766720:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201456.527574:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442766976:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201456.527583:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442766592:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201456.527593:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442766848:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201457.345934:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201457.633623:0:19784:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 110c: DNE: create striped dir, fail MDT2 ========================================================== 14:37:36 (1788201456) 00010000:00010000:0.0:1788201458.370013:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201459.308766:0:19920:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_110c needs >= 2 MDTs 00010000:00010000:0.0:1788201459.394039:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201460.428454:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201461.442442:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201461.589972:0:20073:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 110d: DNE: create striped dir, fail MDT2 and client ========================================================== 14:37:39 (1788201459) 00000100:00100000:1.0:1788201461.634457:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442767104:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201461.634560:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442767360:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201461.634924:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788201461.634935:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442767232:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201461.634970:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442767488:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201461.647409:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442767360:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201461.682230:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442767232:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201461.682414:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442767488:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201461.682425:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442767104:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201462.467113:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201463.390415:0:20209:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_110d needs >= 2 MDTs 00010000:00010000:0.0:1788201463.492422:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201464.515320:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201465.296654:0:20361:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 14:37:43 (1788201463) 00010000:00010000:0.0:1788201465.543012:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201466.562862:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201466.754660:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442767616:192.168.201.130@tcp:400:kworker.0 00000100:00080000:1.0:1788201466.755874:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:02000000:1.0:1788201466.756009:0:38:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff902350a11000: disconnect after 24s idle 00000100:00100000:0.0:1788201466.757118:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442767744:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201466.757208:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201466.757216:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442767872:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201466.764344:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442768000:192.168.201.130@tcp:9:kworker.0 00000100:00100000:0.0:1788201466.771423:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442767872:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201466.778689:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442767616:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201466.778756:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442767744:192.168.201.130@tcp:400:kworker.0 00000100:00080000:0.0:1788201466.779059:0:2358:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff90234519c380 x1875064442768000/t0(0) o9->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201476 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201466.779070:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:0.0:1788201466.779075:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442768000:192.168.201.130@tcp:9:kworker.0 00000001:02000400:0.0:1788201467.080656:0:20498:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_110e needs >= 2 MDTs 00010000:00010000:1.0:1788201467.585233:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201468.609133:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201469.062191:0:20640:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_110f skipping excluded test 110f 00010000:00010000:1.0:1788201469.633811:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201470.657119:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201471.086600:0:20789:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 14:37:49 (1788201469) 00010000:00010000:1.0:1788201471.690972:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201471.883516:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442768128:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201471.883612:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442768384:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201471.883648:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201471.883655:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442768256:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201471.909978:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442768256:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201471.916185:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442768128:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201471.916210:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442768384:192.168.201.130@tcp:400:kworker.0 00010000:00010000:1.0:1788201472.705471:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201472.760393:0:20925:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_110g needs >= 2 MDTs 00010000:00010000:0.0:1788201473.729205:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201474.739534:0:21078:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 111a: DNE: unlink striped dir, fail MDT1 ========================================================== 14:37:53 (1788201473) 00010000:00010000:3.0:1788201474.757780:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201475.777837:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201476.538043:0:21214:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_111a needs >= 2 MDTs 00010000:00010000:1.0:1788201476.802555:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201477.004481:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442768512:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201477.004572:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442768768:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201477.004606:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201477.004612:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442768640:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201477.017518:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442768640:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201477.017536:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442768512:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201477.017547:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442768768:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201477.825133:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201478.243538:0:21366:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 111b: DNE: unlink striped dir, fail MDT2 ========================================================== 14:37:57 (1788201477) 00010000:00010000:0.0:1788201478.849200:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201479.873261:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201479.889514:0:21502:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_111b needs >= 2 MDTs 00010000:00010000:0.0:1788201480.902542:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201481.911562:0:21655:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 14:38:00 (1788201480) 00010000:00010000:1.0:1788201481.921058:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201482.126544:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442768896:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201482.126643:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442769152:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201482.126689:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201482.126696:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442769024:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201482.138228:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442769152:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201482.146149:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442769024:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201482.146166:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442768896:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201482.945181:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201483.542801:0:21791:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_111c needs >= 2 MDTs 00010000:00010000:3.0:1788201483.969065:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201484.993261:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201485.144718:0:21943:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 14:38:03 (1788201483) 00010000:00010000:0.0:1788201486.018359:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201486.368394:0:22080:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_111d needs >= 2 MDTs 00010000:00010000:1.0:1788201487.041825:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201487.235353:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442769280:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201487.235482:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442769536:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201487.235522:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201487.235528:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442769408:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201487.242176:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442769408:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201487.242197:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442769280:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201487.242209:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442769536:192.168.201.130@tcp:400:kworker.0 00010000:00010000:3.0:1788201488.067578:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201488.734182:0:22232:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 ========================================================== 14:38:07 (1788201487) 00010000:00010000:1.0:1788201489.094529:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201490.113496:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201490.557443:0:22368:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_111e needs >= 2 MDTs 00010000:00010000:0.0:1788201491.138030:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201492.164231:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201492.365856:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442769664:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201492.366522:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442769792:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201492.366960:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442769920:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201492.383248:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442769792:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201492.383356:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442769664:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201492.388154:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442769920:192.168.201.130@tcp:400:kworker.0 00000001:02000400:0.0:1788201492.516247:0:22521:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 ========================================================== 14:38:10 (1788201490) 00010000:00010000:0.0:1788201493.190492:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201494.212639:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201494.333730:0:22657:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_111f needs >= 2 MDTs 00010000:00010000:1.0:1788201495.233103:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201496.263346:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201496.614076:0:22809:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 111g: DNE: unlink striped dir, fail MDT1/MDT2 ========================================================== 14:38:14 (1788201494) 00010000:00010000:3.0:1788201497.281949:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201497.479612:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442770176:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201497.479747:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442770048:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201497.479859:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442770304:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201497.491826:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442770048:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201497.491853:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442770304:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201497.496137:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442770176:192.168.201.130@tcp:400:kworker.0 00000001:02000400:0.0:1788201498.204512:0:22946:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_111g needs >= 2 MDTs 00010000:00010000:2.0:1788201498.313984:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201499.332221:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201500.353231:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201500.754495:0:23098:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 112a: DNE: cross MDT rename, fail MDT1 ========================================================== 14:38:18 (1788201498) 00010000:00010000:3.0:1788201501.378003:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201502.405148:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201502.602992:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442770432:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201502.603140:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442770688:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201502.603183:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201502.603866:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442770560:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201502.608365:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442770560:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201502.608389:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442770432:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201502.608401:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442770688:192.168.201.130@tcp:400:kworker.0 00000001:02000400:3.0:1788201502.781864:0:23235:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_112a needs >= 4 MDTs 00010000:00010000:1.0:1788201503.436212:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201504.449299:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201504.691156:0:23387:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 112b: DNE: cross MDT rename, fail MDT2 ========================================================== 14:38:23 (1788201503) 00010000:00010000:1.0:1788201505.480662:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201506.426844:0:23523:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_112b needs >= 4 MDTs 00010000:00010000:1.0:1788201506.499301:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201507.528447:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201507.724842:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442770816:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201507.724973:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442771072:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201507.725006:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201507.725013:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442770944:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201507.740056:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442770816:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201507.740081:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442771072:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201507.740251:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442770944:192.168.201.130@tcp:400:kworker.0 00000001:02000400:2.0:1788201508.451190:0:23675:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 112c: DNE: cross MDT rename, fail MDT3 ========================================================== 14:38:26 (1788201506) 00010000:00010000:1.0:1788201508.546096:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201509.571562:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201510.069588:0:23812:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_112c needs >= 4 MDTs 00010000:00010000:3.0:1788201510.595125:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201511.617258:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201511.854743:0:23964:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 112d: DNE: cross MDT rename, fail MDT4 ========================================================== 14:38:30 (1788201510) 00010000:00010000:1.0:1788201512.648572:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201512.837456:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442771200:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201512.837548:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442771456:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201512.837584:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201512.837590:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442771328:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201512.851665:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442771456:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201512.855108:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442771328:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201512.855137:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442771200:192.168.201.130@tcp:400:kworker.0 00000001:02000400:1.0:1788201513.446704:0:24100:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_112d needs >= 4 MDTs 00010000:00010000:2.0:1788201513.666108:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201514.698000:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201515.111244:0:24253:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 112e: DNE: cross MDT rename, fail MDT1 and MDT2 ========================================================== 14:38:33 (1788201513) 00010000:00010000:3.0:1788201515.714964:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201516.579253:0:24389:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_112e needs >= 4 MDTs 00010000:00010000:1.0:1788201516.737104:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201517.762472:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201517.953443:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442771584:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201517.953554:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442771840:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201517.953594:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201517.953600:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442771712:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201517.976732:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442771712:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201517.976750:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442771584:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201517.976761:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442771840:192.168.201.130@tcp:400:kworker.0 00000001:02000400:3.0:1788201518.257586:0:24541:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3 ========================================================== 14:38:36 (1788201516) 00010000:00010000:1.0:1788201518.792472:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201519.704155:0:24677:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_112f needs >= 4 MDTs 00010000:00010000:2.0:1788201519.815139:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201520.835191:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201521.352264:0:24830:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 112g: DNE: cross MDT rename, fail MDT1 and MDT4 ========================================================== 14:38:40 (1788201520) 00010000:00010000:2.0:1788201521.863985:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201522.881975:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201523.076533:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442771968:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201523.076633:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442772224:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201523.076667:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201523.076674:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442772096:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201523.086277:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442772096:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201523.086295:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442771968:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201523.090434:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442772224:192.168.201.130@tcp:400:kworker.0 00000001:02000400:0.0:1788201523.106649:0:24966:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_112g needs >= 4 MDTs 00010000:00010000:2.0:1788201523.914124:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201524.933769:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201525.081186:0:25118:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 112h: DNE: cross MDT rename, fail MDT2 and MDT3 ========================================================== 14:38:43 (1788201523) 00010000:00010000:0.0:1788201525.958661:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201526.758453:0:25254:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_112h needs >= 4 MDTs 00010000:00010000:1.0:1788201526.977659:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201528.001078:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201528.196354:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442772480:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201528.196460:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:0.0:1788201528.196467:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442772352:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201528.196492:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442772608:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201528.207677:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442772352:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201528.207701:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442772608:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201528.207713:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442772480:192.168.201.130@tcp:400:kworker.0 00000001:02000400:2.0:1788201528.976930:0:25407:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 112i: DNE: cross MDT rename, fail MDT2 and MDT4 ========================================================== 14:38:47 (1788201527) 00010000:00010000:1.0:1788201529.028708:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201530.049136:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201530.823912:0:25543:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_112i needs >= 4 MDTs 00010000:00010000:1.0:1788201531.082361:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201532.099168:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201532.719531:0:25695:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 112j: DNE: cross MDT rename, fail MDT3 and MDT4 ========================================================== 14:38:51 (1788201531) 00010000:00010000:0.0:1788201533.122813:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201533.318472:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442772864:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201533.318611:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788201533.318621:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442772736:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201533.318656:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442772992:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201533.321523:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442772992:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201533.330199:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442772736:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201533.330226:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442772864:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201534.153962:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201534.171725:0:25831:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_112j needs >= 4 MDTs 00010000:00010000:0.0:1788201535.169985:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201535.619649:0:25984:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3 ========================================================== 14:38:54 (1788201534) 00010000:00010000:0.0:1788201536.195002:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201537.221785:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201537.279325:0:26120:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_112k needs >= 4 MDTs 00010000:00010000:3.0:1788201538.241011:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201538.436302:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442773120:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201538.436394:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442773376:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201538.436429:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201538.436436:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442773248:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201538.445454:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442773248:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201538.445474:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442773376:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201538.446508:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442773120:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201539.266315:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201539.283780:0:26272:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 112l: DNE: cross MDT rename, fail MDT1,MDT2,MDT4 ========================================================== 14:38:57 (1788201537) 00010000:00010000:0.0:1788201540.292477:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201541.194035:0:26408:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_112l needs >= 4 MDTs 00010000:00010000:3.0:1788201541.313724:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201542.337132:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201542.970501:0:26561:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 112m: DNE: cross MDT rename, fail MDT1,MDT3,MDT4 ========================================================== 14:39:01 (1788201541) 00010000:00010000:1.0:1788201543.374327:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201543.556761:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442773504:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201543.556924:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442773760:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201543.556968:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201543.556975:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442773632:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201543.568240:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442773632:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201543.574004:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442773504:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201543.574038:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442773760:192.168.201.130@tcp:400:kworker.0 00010000:00010000:1.0:1788201544.396056:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201544.936072:0:26698:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_112m needs >= 4 MDTs 00010000:00010000:3.0:1788201545.409141:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201546.433135:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201546.721244:0:26850:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 112n: DNE: cross MDT rename, fail MDT2,MDT3,MDT4 ========================================================== 14:39:05 (1788201545) 00010000:00010000:1.0:1788201547.465062:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201548.487100:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201548.678453:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442773888:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201548.679919:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442774144:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201548.679972:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201548.679980:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442774016:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201548.688176:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442774144:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201548.691692:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442774016:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201548.691714:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442773888:192.168.201.130@tcp:400:kworker.0 00000001:02000400:0.0:1788201548.720855:0:26987:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_112n needs >= 4 MDTs 00010000:00010000:0.0:1788201549.512463:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201550.537311:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201550.552631:0:27139:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 115: failover for create/unlink striped directory ========================================================== 14:39:09 (1788201549) 00010000:00010000:2.0:1788201551.559837:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201551.996169:0:27275:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_115 needs >= 2 MDTs 00010000:00010000:2.0:1788201552.582145:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201553.602547:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201553.630346:0:27428:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 116a: large update log master MDT recovery ========================================================== 14:39:12 (1788201552) 00000100:00100000:1.0:1788201553.803948:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442774272:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201553.804047:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442774528:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201553.804082:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201553.804089:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442774400:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201553.817494:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442774272:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201553.831057:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442774400:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201553.831076:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442774528:192.168.201.130@tcp:400:kworker.0 00010000:00010000:3.0:1788201554.625000:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201555.240651:0:27564:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_116a needs >= 2 MDTs 00010000:00010000:3.0:1788201555.649686:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201556.675214:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201557.294764:0:27716:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 116b: large update log slave MDT recovery ========================================================== 14:39:15 (1788201555) 00010000:00010000:1.0:1788201557.700985:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201558.722448:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201558.915654:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442774656:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201558.915757:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442774912:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201558.915789:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201558.915795:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442774784:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201558.920226:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442774784:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201558.922982:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442774912:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201558.929578:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442774656:192.168.201.130@tcp:400:kworker.0 00000001:02000400:1.0:1788201559.005840:0:27852:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_116b needs >= 2 MDTs 00010000:00010000:2.0:1788201559.750387:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201560.769140:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201560.818713:0:28005:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 117: DNE: cross MDT unlink, fail MDT1 and MDT2 ========================================================== 14:39:19 (1788201559) 00010000:00010000:1.0:1788201561.793236:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201562.595779:0:28141:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_117 needs >= 4 MDTs 00010000:00010000:3.0:1788201562.817002:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201563.843205:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201564.034848:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442775040:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201564.039452:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201564.039476:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442775168:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201564.041475:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201564.041496:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442775296:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201564.048822:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442775168:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201564.048842:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442775040:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201564.049029:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442775296:192.168.201.130@tcp:400:kworker.0 00000001:02000400:3.0:1788201564.723466:0:28293:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 118: invalidate osp update will not cause update log corruption ========================================================== 14:39:23 (1788201563) 00010000:00010000:3.0:1788201564.866145:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201565.900984:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201566.549746:0:28430:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_118 needs >= 2 MDTs 00010000:00010000:3.0:1788201566.912961:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201567.937283:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201568.902290:0:28582:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 119: timeout of normal replay does not cause DNE replay fails ========================================================== 14:39:27 (1788201567) 00010000:00010000:3.0:1788201568.964687:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201569.165127:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442775424:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201569.165233:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442775680:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201569.165267:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201569.165273:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442775552:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201569.171241:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442775552:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201569.171259:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442775680:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201569.185741:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442775424:192.168.201.130@tcp:400:kworker.0 00010000:00010000:3.0:1788201569.984989:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788201570.666086:0:28718:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_119 needs >= 2 MDTs 00010000:00010000:3.0:1788201571.011096:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201572.035137:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201573.038696:0:28871:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 120: DNE fail abort should stop both normal and DNE replay ========================================================== 14:39:31 (1788201571) 00010000:00010000:3.0:1788201573.057002:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201574.081215:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201574.276827:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442775936:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201574.276970:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788201574.276978:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442775808:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201574.277010:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442776064:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201574.286110:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442775808:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201574.286134:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442776064:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201574.286145:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442775936:192.168.201.130@tcp:400:kworker.0 00000001:02000400:3.0:1788201574.435721:0:29007:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: SKIP: replay-single test_120 needs >= 2 MDTs 00010000:00010000:3.0:1788201575.105120:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201576.081204:0:29159:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 121: lock replay timed out and race ========================================================== 14:39:34 (1788201574) 00010000:00010000:0.0:1788201576.130268:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201577.153071:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788201577.545425:0:29183:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201577.545466:0:29183:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201577.545492:0:29183:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 97 lru_type: 1 00010000:00010000:3.0:1788201577.545511:0:29183:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 97 lru_type: 1 00000080:00200000:3.0:1788201577.545520:0:29183:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201577.545530:0:29183:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201577.545535:0:29183:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:3.0:1788201577.545544:0:29183:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181077bf8 00010000:00010000:3.0:1788201577.545550:0:29183:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 97 lru_type: 1 00010000:00010000:3.0:1788201577.545560:0:29183:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 98 lru_type: 1 00000080:00200000:3.0:1788201577.545582:0:29183:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201577.545601:0:29183:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201577.545606:0:29183:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201577.545613:0:29183:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 98 lru_type: 1 00010000:00010000:3.0:1788201577.545623:0:29183:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 98 lru_type: 1 00000080:00200000:3.0:1788201577.545635:0:29183:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201577.545639:0:29183:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201577.545642:0:29183:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5365 00000080:00010000:3.0:1788201577.545645:0:29183:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5365 from it ffffa0b181077b38 00010000:00010000:3.0:1788201577.545649:0:29183:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 98 lru_type: 1 00010000:00010000:3.0:1788201577.545656:0:29183:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 99 lru_type: 1 00000080:00200000:3.0:1788201577.545664:0:29183:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 3 00000080:00200000:3.0:1788201577.545670:0:29183:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f121.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), file ffff90235179f680, open_flags 8941, mode 81b6 opened 0 00000080:00200000:3.0:1788201577.545682:0:29183:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f121.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=open|creat 00000002:00010000:3.0:1788201577.545708:0:29183:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f121.replay-single,[0x200000bd1:0x429:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0104502 00010000:00010000:3.0:1788201577.545722:0:29183:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 99 lru_type: 1 00010000:00010000:3.0:1788201577.545785:0:29183:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 99 lru_type: 1 00010000:00010000:3.0:1788201577.545793:0:29183:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201577.545797:0:29183:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c5365 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d5e2b expref: -99 pid: 17857 timeout: 0 lvb_type: 0 lru_score: 99 lru_type: 1 00010000:00010000:3.0:1788201577.545846:0:29183:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 29183 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201577.545859:0:29183:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 29183 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201577.545873:0:29183:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201577.545912:0:29183:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352688a80 x1875064442776320/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201577.545923:0:29183:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 29183 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201577.545952:0:29183:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29183:1875064442776320:192.168.201.130@tcp:101:touch.0 00000100:00100000:3.0:1788201577.551380:0:29183:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fa600 going to sleep for 11 seconds 00000100:00100000:3.0:1788201577.580243:0:29183:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29183:1875064442776320:192.168.201.130@tcp:101:touch.0 00010000:00010000:3.0:1788201577.580273:0:29183:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5feb expref: -99 pid: 29183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201577.580306:0:29183:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5feb expref: -99 pid: 29183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201577.580314:0:29183:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5feb expref: -99 pid: 29183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201577.580326:0:29183:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5feb expref: -99 pid: 29183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201577.580337:0:29183:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5feb expref: -99 pid: 29183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201577.580354:0:29183:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5feb expref: -99 pid: 29183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201577.580362:0:29183:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5feb expref: -99 pid: 29183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201577.580369:0:29183:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d5feb expref: -99 pid: 29183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201577.580382:0:29183:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff902352688a80 x1875064442776320/t21474836481(21474836481) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201593 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201577.580394:0:29183:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902352688a80 x1875064442776320/t21474836481(21474836481) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201593 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201577.580408:0:29183:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5feb expref: -99 pid: 29183 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201577.580420:0:29183:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d5feb expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201577.580430:0:29183:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1065 (0 0) 00000080:00200000:3.0:1788201577.580437:0:29183:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201577.580451:0:29183:0:(namei.c:1628:ll_create_it()) VFS Op:name=f121.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=open|creat 00000080:00200000:3.0:1788201577.580478:0:29183:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x429:0x0](ffff90234497d348) 00000080:00200000:3.0:1788201577.580597:0:29183:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x429:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201577.580608:0:29183:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x429:0x0](ffff90234497d348) 00000080:00010000:3.0:1788201577.580612:0:29183:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x429:0x0](ffff90234497d348) for lock 0x4605c7c97e1c58d6 00000080:00010000:3.0:1788201577.580621:0:29183:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x429:0x0](ffff90234497d348) 00000080:00200008:3.0:1788201577.580634:0:29183:0:(file.c:1090:ll_file_open()) START file f121.replay-single:[0x200000bd1:0x429:0x0](ffff90234497d348), flags 104501 00000002:00100000:3.0:1788201577.580652:0:29183:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff902352688a80 x1875064442776320/t21474836481(21474836481) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201593 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201577.580690:0:29183:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c58d6 from it ffff9023533ea3c0 00010000:00010000:3.0:1788201577.580695:0:29183:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d5feb expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201577.580704:0:29183:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d5feb expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201577.580724:0:29183:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x429:0x0](ffff90234497d348), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201577.580757:0:29183:0:(llite_lib.c:2436:ll_setattr_raw()) START file f121.replay-single:[0x200000bd1:0x429:0x0](ffff90234497d348) current size 0, valid attrs 20070, mode 9023, uid -2130215344, gid -24399, new size 18446744071973566999, atime 1788201577.000000000, mtime 1788201577.000000000, ctime 1788201577.000000000 00010000:00010000:3.0:1788201577.580774:0:29183:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d5feb expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201577.580794:0:29183:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x429:0x0]: losing layout lock 00010000:00010000:3.0:1788201577.580830:0:29183:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5feb expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201577.580837:0:29183:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201577.580840:0:29183:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58d6 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d5feb expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788201577.580861:0:29183:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201577.580868:0:29183:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90235268aa00 x1875064442776576/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201577.580955:0:29183:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29183:1875064442776576:192.168.201.130@tcp:36:touch.0 00000100:00100000:3.0:1788201577.581044:0:29183:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fae40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201577.588350:0:29183:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29183:1875064442776576:192.168.201.130@tcp:36:touch.0 00000080:00200000:3.0:1788201577.588437:0:29183:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x429:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:3.0:1788201577.588459:0:29183:0:(file.c:7046:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000bd1:0x429:0x0](ffff90234497d348) 00000002:00010000:3.0:1788201577.588477:0:29183:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x429:0x0]) in obj [0x200000bd1:0x429:0x0], intent: layout flags 0 00010000:00010000:3.0:1788201577.588532:0:29183:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58dd lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x429:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 29183 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201577.588543:0:29183:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58dd lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x429:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 29183 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201577.588554:0:29183:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58dd lrc: 3/1,0 mode: --/CR res: [0x200000bd1:0x429:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 29183 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201577.588575:0:29183:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29183:1875064442776704:192.168.201.130@tcp:101:touch.0 00000100:00100000:3.0:1788201577.588655:0:29183:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fae40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201577.598232:0:29183:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29183:1875064442776704:192.168.201.130@tcp:101:touch.0 00010000:00010000:3.0:1788201577.598262:0:29183:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58dd lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x429:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6000 expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201577.598276:0:29183:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58dd lrc: 4/1,0 mode: CR/CR res: [0x200000bd1:0x429:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6000 expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201577.598291:0:29183:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff902352689c00 x1875064442776704/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 376/400 e 0 to 0 dl 1788201593 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201577.598309:0:29183:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58dd lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x429:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6000 expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201577.598323:0:29183:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58dd lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x429:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6000 expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201577.598336:0:29183:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 16 res 8589937617/1065 (0 0) 00000080:00200000:3.0:1788201577.598342:0:29183:0:(namei.c:952:ll_intent_lock()) intent lock 1024 on i1 [0x200000bd1:0x429:0x0] suppgids 0 0: rc 0 00000080:00010000:3.0:1788201577.598359:0:29183:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x429:0x0](ffff90234497d348) for lock 0x4605c7c97e1c58dd 00000080:00010000:3.0:1788201577.598366:0:29183:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c58dd from it ffffa0b181077a78 00010000:00010000:3.0:1788201577.598371:0:29183:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58dd lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x429:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6000 expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201577.598382:0:29183:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58dd lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x429:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6000 expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201577.598404:0:29183:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58dd lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x429:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6000 expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201577.598416:0:29183:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58dd lrc: 2/1,0 mode: CR/CR res: [0x200000bd1:0x429:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6000 expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201577.598424:0:29183:0:(file.c:6929:ll_layout_lock_set()) ### file [0x200000bd1:0x429:0x0](00000000e95b4195) being reconfigured ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58dd lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x429:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6000 expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201577.598483:0:29183:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x429:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:3.0:1788201577.598502:0:29183:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58dd lrc: 3/1,0 mode: CR/CR res: [0x200000bd1:0x429:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d6000 expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201577.598511:0:29183:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58dd lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x429:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d6000 expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:3.0:1788201577.598531:0:29183:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:3.0:1788201577.598619:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201577.598633:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442776832:192.168.201.130@tcp:2:touch.0 00000100:00100000:3.0:1788201577.626628:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442776832:192.168.201.130@tcp:2:touch.0 00000080:00200000:2.0:1788201577.626738:0:29183:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x429:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:2.0:1788201577.626760:0:29183:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file f121.replay-single:[0x200000bd1:0x429:0x0](ffff90234497d348) current size 0, valid attrs 100201f0, mode 9023, uid -2130215344, gid -24399, new size 18446744071973566999, atime 1788201577.000000000, mtime 1788201577.000000000, ctime 1788201577.000000000, rc 0 00000080:00200008:2.0:1788201577.626787:0:29183:0:(file.c:479:ll_file_release()) START file f121.replay-single:[0x200000bd1:0x429:0x0](ffff90234497d348), flags 104001 00000002:00100000:2.0:1788201577.626842:0:29183:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff902352688a80 x1875064442776320/t21474836481(21474836481) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201593 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201577.626853:0:29183:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201577.626858:0:29183:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234598e680 x1875064442777088/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201577.626915:0:29183:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29183:1875064442777088:192.168.201.130@tcp:35:touch.0 00000100:00100000:2.0:1788201577.626994:0:29183:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234502f9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201577.634279:0:29183:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29183:1875064442777088:192.168.201.130@tcp:35:touch.0 00000002:00100000:3.0:1788201577.634326:0:29183:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff902352688a80 x1875064442776320/t21474836481(21474836481) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201593 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1788201577.703039:0:29185:0:(ldlm_resource.c:195:lru_size_store()) dropping all unused locks from namespace lustre-MDT0000-mdc-ffff902350a11000 00010000:00010000:3.0:1788201577.703189:0:29185:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1788201577.703866:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201577.704180:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58dd lrc: 2/0,0 mode: CR/CR res: [0x200000bd1:0x429:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d6000 expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201577.704260:0:13084:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x429:0x0]: losing layout lock 00010000:00010000:1.0:1788201577.704342:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58dd lrc: 1/0,0 mode: --/CR res: [0x200000bd1:0x429:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d6000 expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201577.704351:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788201577.704384:0:13084:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_03:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13084:1875064442777216:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788201577.704577:0:13084:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342e36180 going to sleep for 11 seconds 00000100:00100000:1.0:1788201577.728321:0:13084:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ldlm_bl_03:35604eb4-baf9-45be-a20a-4883a2cd8cf9:13084:1875064442777216:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201577.728361:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684a00/0x4605c7c97e1c58dd lrc: 0/0,0 mode: --/CR res: [0x200000bd1:0x429:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d6000 expref: -99 pid: 29183 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201577.728455:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201577.728458:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201577.728461:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788201577.977825:0:29189:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201577.977867:0:29189:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201577.977965:0:29189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 29189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201577.977978:0:29189:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 29189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201577.977991:0:29189:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 29189 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201577.978035:0:29189:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job multiop:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29189:1875064442777344:192.168.201.130@tcp:101:multiop.0 00000100:00100000:3.0:1788201577.978322:0:29189:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023436fae40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201577.984634:0:29189:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job multiop:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29189:1875064442777344:192.168.201.130@tcp:101:multiop.0 00010000:00010000:0.0:1788201577.984671:0:29189:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d600e expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.984688:0:29189:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d600e expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.984701:0:29189:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d600e expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.984714:0:29189:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d600e expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.984729:0:29189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d600e expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.984745:0:29189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d600e expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.984753:0:29189:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d600e expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201577.984765:0:29189:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff90235268a680 x1875064442777344/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201593 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201577.984781:0:29189:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d600e expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.984796:0:29189:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201577.984805:0:29189:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201577.984843:0:29189:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:0.0:1788201577.984850:0:29189:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c58e4 00000080:00010000:0.0:1788201577.984856:0:29189:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201577.984860:0:29189:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c58e4 from it ffffa0b180c87b58 00010000:00010000:0.0:1788201577.984863:0:29189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d600e expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.984873:0:29189:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d600e expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201577.984938:0:29189:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201577.984964:0:29189:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f121.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=9 00000080:00200000:0.0:1788201577.984970:0:29189:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f121.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=getattr 00000002:00010000:0.0:1788201577.985009:0:29189:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f121.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201577.985064:0:29189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 29189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.985072:0:29189:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 29189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.985083:0:29189:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 29189 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201577.985109:0:29189:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job multiop:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29189:1875064442777472:192.168.201.130@tcp:101:multiop.0 00000100:00100000:0.0:1788201577.985214:0:29189:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005a780 going to sleep for 11 seconds 00000100:00100000:0.0:1788201577.988761:0:29189:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job multiop:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29189:1875064442777472:192.168.201.130@tcp:101:multiop.0 00010000:00010000:0.0:1788201577.988795:0:29189:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.988822:0:29189:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.988832:0:29189:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.988845:0:29189:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.988857:0:29189:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.988872:0:29189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.988930:0:29189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.988938:0:29189:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201577.988951:0:29189:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023450b9c00 x1875064442777472/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201593 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201577.988964:0:29189:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201577.988979:0:29189:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.989000:0:29189:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1065 (0 0) 00000080:00200000:0.0:1788201577.989012:0:29189:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201577.989143:0:29189:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x429:0x0](ffff90234497d348): rc = 0 00000080:00200000:0.0:1788201577.989168:0:29189:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x429:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201577.989181:0:29189:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x429:0x0](ffff90234497d348) for lock 0x4605c7c97e1c58eb 00000080:00010000:0.0:1788201577.989196:0:29189:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x429:0x0](ffff90234497d348) 00000080:00010000:0.0:1788201577.989200:0:29189:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c58eb from it ffffa0b180c87b20 00010000:00010000:0.0:1788201577.989203:0:29189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.989213:0:29189:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201577.989266:0:29189:0:(file.c:6283:ll_getattr_dentry()) START file f121.replay-single:[0x200000bd1:0x429:0x0](ffff90234497d348), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201577.989275:0:29189:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x429:0x0](ffff90234497d348),name=f121.replay-single 00000002:00010000:0.0:1788201577.989299:0:29189:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x429:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201577.989310:0:29189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201577.989322:0:29189:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201577.989329:0:29189:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201577.989333:0:29189:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x429:0x0](ffff90234497d348) 00000080:00010000:0.0:1788201577.989337:0:29189:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x429:0x0](ffff90234497d348) for lock 0x4605c7c97e1c58eb 00000080:00010000:0.0:1788201577.989340:0:29189:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c58eb from it ffffa0b180c87cb8 00010000:00010000:0.0:1788201577.989343:0:29189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201577.989350:0:29189:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201577.989378:0:29189:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x429:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:0.0:1788201577.989389:0:29189:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x429:0x0] 00000008:00010000:0.0:1788201577.989427:0:29189:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:0.0:1788201577.989432:0:29189:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff902350050530, osc lock ffff902343fb6f70, flags 0x400001002 00010000:00010000:0.0:1788201577.989493:0:29189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.989512:0:29189:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 3/1,0 mode: --/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788201577.989560:0:29189:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1788201577.989580:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201577.989600:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442777600:192.168.201.130@tcp:101:multiop.0 00010000:00010000:0.0:1788201577.996869:0:2359:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.996954:0:2359:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 6/2,0 mode: --/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.996974:0:2359:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.996990:0:2359:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1788201577.997019:0:2359:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 5/2,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:0.0:1788201577.997033:0:2359:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:0.0:1788201577.997056:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 6/2,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.997072:0:2359:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.997079:0:2359:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201577.997089:0:2359:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:0.0:1788201577.997114:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442777600:192.168.201.130@tcp:101:multiop.0 00000080:00200000:3.0:1788201577.997232:0:29189:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x429:0x0] updating i_size 0 i_blocks 1 00000020:00010000:3.0:1788201577.997241:0:29189:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902346372750 (ffff902348a2f638/0) at cl_lock_release():245 00000020:00010000:3.0:1788201577.997245:0:29189:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902346372750 (ffff902348a2f638/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788201577.997251:0:29189:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350050530 (ffff902348a2f008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788201577.997256:0:29189:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902346372750 (ffff902348a2f638/0) at cl_lock_fini():72 00000020:00010000:3.0:1788201577.997259:0:29189:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350050530 (ffff902348a2f638/1) at cl_lock_fini():72 00000080:00200000:3.0:1788201577.997313:0:29189:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x429:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:3.0:1788201578.178808:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201579.204039:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201579.393701:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442777728:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201579.393841:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442777984:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201579.393923:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201579.393932:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442777856:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201579.406933:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442777856:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201579.406963:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442777984:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201579.408262:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442777728:192.168.201.130@tcp:400:kworker.0 00010000:00010000:3.0:1788201580.225262:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201581.249374:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201582.273757:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201583.297212:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201584.321230:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201584.515018:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442778112:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201584.515155:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442778368:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201584.515197:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788201584.515204:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442778240:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201584.531800:0:2361:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0000-osc-ffff902350a11000: committing for last_committed 4294968892 gen 2 00000100:00100000:3.0:1788201584.531843:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442778368:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201585.350468:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201586.372935:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201587.395911:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201588.418516:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201589.447283:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201589.633167:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442778496:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201589.633242:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442778624:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201589.633306:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442778752:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201589.652583:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442778752:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201590.465145:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201591.489908:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201592.514032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201593.541100:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201594.561938:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201594.753639:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442779008:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201594.753663:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442778880:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201594.753817:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442779136:192.168.201.130@tcp:400:kworker.0 00000100:00080000:0.0:1788201594.770182:0:2360:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff902350a11000 of lustre-MDT0000_UUID@192.168.201.130@tcp abruptly disconnected: reconnecting 00000100:00080000:0.0:1788201594.770191:0:2360:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1788201594.770196:0:2360:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff902350a11000: Connection to lustre-MDT0000 (at 192.168.201.130@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1788201594.772455:0:2359:0:(recover.c:205:ptlrpc_request_handle_notconn()) import MGC192.168.201.130@tcp of MGS@192.168.201.130@tcp abruptly disconnected: reconnecting 00000100:00080000:1.0:1788201594.772463:0:2359:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from FULL to DISCONN 00000100:02020000:1.0:1788201594.772468:0:2359:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail 00000100:00080000:0.0:1788201594.778041:0:2360:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201594.778055:0:2360:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 698 00000100:00080000:0.0:1788201594.778071:0:2360:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201594.778083:0:2360:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201594.778119:0:2360:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442779264/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201594.778322:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442779264:192.168.201.130@tcp:38:ptlrpcd_00_01.0 00000100:00100000:0.0:1788201594.778462:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442779008:192.168.201.130@tcp:400:kworker.0 00000100:00080000:1.0:1788201594.785914:0:2359:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.201.130@tcp for MGC192.168.201.130@tcp not replayable, auto-deactivating 00000100:00080000:1.0:1788201594.785921:0:2359:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:1.0:1788201594.785931:0:2359:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff90235268bb80 x1875064442778112/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201600 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201594.786029:0:2359:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff902352688700 x1875064442778496/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201605 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201594.786055:0:2359:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff9023450b9c00 x1875064442778880/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201610 ref 1 fl Rpc:RNQU/200/0 rc 0/-107 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:1.0:1788201594.786064:0:2359:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:1.0:1788201594.786067:0:2359:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:1.0:1788201594.786069:0:2359:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:1.0:1788201594.786074:0:2359:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201594.786081:0:2359:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 688 00000100:00080000:1.0:1788201594.786096:0:2359:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788201594.786129:0:2359:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442779392/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201594.786169:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442778880:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201594.786211:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442779136:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201594.786355:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442779392:192.168.201.130@tcp:250:ptlrpcd_00_00.0 00000100:00100000:3.0:1788201594.786671:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442778496:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201594.787502:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442778112:192.168.201.130@tcp:400:kworker.0 00000100:00080000:0.0:1788201594.800549:0:2358:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.201.130@tcp: connect to target with instance 0 10000000:01000000:0.0:1788201594.800564:0:2358:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:0.0:1788201594.800570:0:2358:0:(import.c:1284:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x9383f985bc4d5cdb to 0x9383f985bc4d61ce 00000100:00080000:0.0:1788201594.810488:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:0.0:1788201594.810502:0:2358:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.201.130@tcp; invalidating 00000100:00100000:0.0:1788201594.811000:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442779392:192.168.201.130@tcp:250:ptlrpcd_00_00.0 00010000:00080000:0.0:1788201594.811039:0:2358:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff9023450b9180 x1875064442779264/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/416 e 0 to 0 dl 1788201604 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ptlrpcd_00_01.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201594.811051:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 21474836483 gen 2 00000100:00080000:0.0:1788201594.811079:0:2358:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff902350a11000: connect to target with instance 6 00000100:00080000:0.0:1788201594.811096:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:0.0:1788201594.811099:0:2358:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788201594.811100:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 21474836483 gen 2 00000100:00080000:0.0:1788201594.811102:0:2358:0:(recover.c:53:ptlrpc_replay_next()) import ffff902348897000 from lustre-MDT0000_UUID committed 21474836483 last 0 00000100:00080000:0.0:1788201594.811105:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:2.0:1788201594.811370:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442779264:192.168.201.130@tcp:38:ptlrpcd_00_01.0 00010000:00080000:0.0:1788201594.811416:0:29503:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff902350a11000 to lustre-MDT0000_UUID@192.168.201.130@tcp 00010000:00010000:0.0:1788201594.811861:0:29503:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00080000:0.0:1788201594.811899:0:29503:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:0.0:1788201594.811905:0:29503:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d600e expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00080000:0.0:1788201594.811911:0:29503:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00000100:00100000:2.0:1788201594.812029:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442779520:192.168.201.130@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788201594.812091:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442779648:192.168.201.130@tcp:101:ldlm_lock_repla.0 00000100:00080000:3.0:1788201594.813065:0:29502:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.201.130@tcp to MGS@192.168.201.130@tcp 00000100:00100000:3.0:1788201594.813073:0:29502:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:3.0:1788201594.813079:0:29502:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:3.0:1788201594.813097:0:29502:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff90235265c000/0x4605c7c97e1c58b3 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0x9383f985bc4d5ce9 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788201594.813110:0:29502:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff90235265c000/0x4605c7c97e1c58b3 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0x9383f985bc4d5ce9 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201594.813117:0:29502:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:3.0:1788201594.813374:0:29502:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201594.813384:0:29502:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff90235265c000/0x4605c7c97e1c58b3 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0x9383f985bc4d5ce9 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201594.813397:0:29502:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff90235265c400/0x4605c7c97e1c58c1 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x9383f985bc4d5cf7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788201594.813402:0:29502:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff90235265c400/0x4605c7c97e1c58c1 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x9383f985bc4d5cf7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201594.813407:0:29502:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:3.0:1788201594.813496:0:29502:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201594.813501:0:29502:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff90235265c400/0x4605c7c97e1c58c1 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x9383f985bc4d5cf7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201594.813516:0:29502:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff902345107800/0x4605c7c97e1c58cf lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x9383f985bc4d5e24 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788201594.813523:0:29502:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff902345107800/0x4605c7c97e1c58cf lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x9383f985bc4d5e24 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201594.813529:0:29502:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788201594.813613:0:29502:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201594.813618:0:29502:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff902345107800/0x4605c7c97e1c58cf lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x9383f985bc4d5e24 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201594.813631:0:29502:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff90235265d600/0x4605c7c97e1c58ba lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x9383f985bc4d5cf0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788201594.813637:0:29502:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff90235265d600/0x4605c7c97e1c58ba lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x9383f985bc4d5cf0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201594.813641:0:29502:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:3.0:1788201594.813890:0:29502:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201594.813894:0:29502:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff90235265d600/0x4605c7c97e1c58ba lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x9383f985bc4d5cf0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1788201594.813908:0:29502:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:3.0:1788201594.813912:0:29502:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from RECOVER to FULL 10000000:01000000:3.0:1788201594.813914:0:29502:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:3.0:1788201594.814502:0:29502:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:3.0:1788201594.814510:0:29502:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.201.130@tcp: Connection restored to 192.168.201.130@tcp (at 192.168.201.130@tcp) 00010000:00010000:0.0:1788201595.585351:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788201595.968972:0:5601:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788201595.968982:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-cliir-ffff902350a11000 from 18 10000000:01000000:2.0:1788201595.968989:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788201595.969045:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff902345106a00/0x4605c7c97e1c58f9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201595.969058:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff902345106a00/0x4605c7c97e1c58f9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201595.969071:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff902345106a00/0x4605c7c97e1c58f9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201595.969100:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442779776:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201595.969202:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234502f0c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201595.976860:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442779776:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201595.976926:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff902345106a00/0x4605c7c97e1c58f9 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d61d5 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201595.976941:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff902345106a00/0x4605c7c97e1c58f9 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d61d5 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201595.977026:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442779904:192.168.201.130@tcp:256:ll_cfg_requeue.0 00000100:00100000:0.0:1788201595.977117:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005a480 going to sleep for 11 seconds 00000100:00100000:0.0:1788201595.989946:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442779904:192.168.201.130@tcp:256:ll_cfg_requeue.0 10000000:01000000:0.0:1788201595.990045:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:0.0:1788201595.990056:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff902345106a00/0x4605c7c97e1c58f9 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d61d5 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201595.990067:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff902345106a00/0x4605c7c97e1c58f9 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d61d5 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201595.990077:0:5601:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:0.0:1788201595.990080:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-client-ffff902350a11000 from 42 10000000:01000000:0.0:1788201595.990083:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:0.0:1788201595.990119:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff90235865be00/0x4605c7c97e1c5900 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201595.990132:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff90235865be00/0x4605c7c97e1c5900 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201595.990141:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff90235865be00/0x4605c7c97e1c5900 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201595.990164:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442780032:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788201595.990250:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005a480 going to sleep for 11 seconds 00000100:00100000:0.0:1788201596.007677:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442780032:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201596.007721:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff90235865be00/0x4605c7c97e1c5900 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d61dc expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201596.007734:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff90235865be00/0x4605c7c97e1c5900 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d61dc expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201596.007825:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442780160:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788201596.007962:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005a480 going to sleep for 11 seconds 00000100:00100000:0.0:1788201596.021266:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442780160:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788201596.021331:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442780288:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000100:00100000:0.0:1788201596.021424:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005a480 going to sleep for 11 seconds 00000100:00100000:0.0:1788201596.031947:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442780288:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000040:00080000:3.0:1788201596.036862:0:29504:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 42 00000020:01000000:0.0:1788201596.037074:0:5601:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 42-41 (rc=0) 10000000:01000000:0.0:1788201596.037093:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:0.0:1788201596.037106:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff90235865be00/0x4605c7c97e1c5900 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d61dc expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201596.037118:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff90235865be00/0x4605c7c97e1c5900 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d61dc expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201596.037128:0:5601:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:0.0:1788201596.037131:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log params-ffff902350a11000 from 7 10000000:01000000:0.0:1788201596.037134:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:0.0:1788201596.037171:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff90235865b000/0x4605c7c97e1c5907 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201596.037179:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff90235865b000/0x4605c7c97e1c5907 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201596.037192:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff90235865b000/0x4605c7c97e1c5907 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201596.037218:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442780416:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788201596.037311:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005a480 going to sleep for 11 seconds 00000100:00100000:0.0:1788201596.049152:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442780416:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201596.049182:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff90235865b000/0x4605c7c97e1c5907 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d61e3 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201596.049196:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff90235865b000/0x4605c7c97e1c5907 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d61e3 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201596.049276:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442780544:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788201596.049582:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005a480 going to sleep for 11 seconds 00000100:00100000:0.0:1788201596.051752:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442780544:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788201596.051817:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442780672:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000100:00100000:0.0:1788201596.052163:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005a480 going to sleep for 11 seconds 00000100:00100000:0.0:1788201596.055766:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442780672:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000040:00080000:2.0:1788201596.057242:0:29505:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000020:01000000:0.0:1788201596.057632:0:5601:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:0.0:1788201596.057652:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:0.0:1788201596.057665:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff90235865b000/0x4605c7c97e1c5907 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d61e3 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201596.057691:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff90235865b000/0x4605c7c97e1c5907 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d61e3 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201596.057702:0:5601:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:0.0:1788201596.057705:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-sptlrpc-ffff902350bf19c0 from 1 10000000:01000000:0.0:1788201596.057708:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:0.0:1788201596.058688:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff90235865a600/0x4605c7c97e1c590e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201596.058703:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff90235865a600/0x4605c7c97e1c590e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201596.058776:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff90235865a600/0x4605c7c97e1c590e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201596.059362:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442780800:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788201596.059465:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005a480 going to sleep for 11 seconds 00000100:00100000:0.0:1788201596.065686:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442780800:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201596.065719:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff90235865a600/0x4605c7c97e1c590e lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d61ea expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201596.065734:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff90235865a600/0x4605c7c97e1c590e lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d61ea expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201596.065844:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442780928:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788201596.065950:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90235005a480 going to sleep for 11 seconds 00000100:00100000:0.0:1788201596.074982:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442780928:192.168.201.130@tcp:501:ll_cfg_requeue.0 10000000:01000000:0.0:1788201596.075031:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:0.0:1788201596.075043:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff90235865a600/0x4605c7c97e1c590e lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d61ea expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201596.075055:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff90235865a600/0x4605c7c97e1c590e lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d61ea expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201596.614141:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201597.633629:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201597.658628:0:2358:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1788201597.658668:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442779648:192.168.201.130@tcp:101:ldlm_lock_repla.0 00010000:00010000:2.0:1788201598.667082:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201599.690773:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201599.875061:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201599.875082:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:1.0:1788201599.875459:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442781056:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201599.875650:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201599.875658:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442781184:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201599.894839:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442781184:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201599.901685:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442781056:192.168.201.130@tcp:400:kworker.0 00010000:00010000:1.0:1788201600.707684:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788201600.770253:0:2361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201584/real 1788201584] req@ffff90235268ad80 x1875064442778240/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201600 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201600.809754:0:2361:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff90235268ad80 x1875064442778240/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201600 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201600.809870:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442778240:192.168.201.130@tcp:400:kworker.0 00000080:00200008:0.0:1788201600.874856:0:29575:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201600.874872:0:29575:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201600.874976:0:29575:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201600.874998:0:29575:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201600.875016:0:29575:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201600.875025:0:29575:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201600.875096:0:29575:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201600.875101:0:29575:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c58e4 00000080:00010000:0.0:1788201600.875109:0:29575:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c58e4 from it ffffa0b180d7fcb8 00010000:00010000:0.0:1788201600.875112:0:29575:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201600.875121:0:29575:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200008:0.0:1788201600.875783:0:29575:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788201600.875788:0:29575:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201600.875796:0:29575:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201600.875805:0:29575:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201600.875814:0:29575:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201600.875820:0:29575:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201600.875824:0:29575:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201600.875827:0:29575:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c58e4 00000080:00010000:0.0:1788201600.875830:0:29575:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c58e4 from it ffffa0b180d7fcb8 00010000:00010000:0.0:1788201600.875833:0:29575:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201600.875845:0:29575:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201600.876153:0:29575:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201600.876162:0:29575:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201600.876169:0:29575:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201600.876178:0:29575:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201600.876185:0:29575:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201600.876189:0:29575:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201600.876192:0:29575:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c58e4 00000080:00010000:0.0:1788201600.876195:0:29575:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c58e4 from it ffffa0b180d7fbb0 00010000:00010000:0.0:1788201600.876198:0:29575:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201600.876207:0:29575:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788201600.876218:0:29575:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 44 00000080:00200008:0.0:1788201600.876225:0:29575:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00000080:00200080:0.0:1788201600.876414:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201600.876428:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00800000:00000004:0.0:1788201600.876434:0:29575:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: age 718, max_age 876 00000100:00080000:0.0:1788201600.876543:0:29575:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != REPLAY_LOCKS) req@ffff9023450ba300 x1875064442781312/t0(0) o41->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201600.876554:0:29575:0:(client.c:2702:ptlrpc_set_wait()) set ffff902350db2a80 going to sleep for 0 seconds 00010000:00010000:2.0:1788201601.732038:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201601.920981:0:29575:0:(client.c:2702:ptlrpc_set_wait()) set ffff902350db2a80 going to sleep for 0 seconds 00010000:00010000:2.0:1788201602.754754:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201602.945045:0:29575:0:(client.c:2702:ptlrpc_set_wait()) set ffff902350db2a80 going to sleep for 0 seconds 00010000:00010000:0.0:1788201603.777057:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201603.969043:0:29575:0:(client.c:2702:ptlrpc_set_wait()) set ffff902350db2a80 going to sleep for 0 seconds 00010000:00010000:1.0:1788201604.801982:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201604.993101:0:29575:0:(client.c:2702:ptlrpc_set_wait()) set ffff902350db2a80 going to sleep for 0 seconds 00000100:00080000:2.0:1788201604.993310:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201604.993315:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:2.0:1788201604.993485:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442781568:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201604.993601:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201604.993608:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442781440:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201605.001146:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442781440:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201605.001171:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442781568:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201605.825457:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1788201605.952993:0:2361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201589/real 1788201589] req@ffff902352689c00 x1875064442778624/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201605 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201605.953034:0:2361:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff902352689c00 x1875064442778624/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201605 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201605.953054:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442778624:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201606.017243:0:29575:0:(client.c:2702:ptlrpc_set_wait()) set ffff902350db2a80 going to sleep for 0 seconds 00010000:00010000:2.0:1788201606.849265:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201607.041919:0:29575:0:(client.c:2702:ptlrpc_set_wait()) set ffff902350db2a80 going to sleep for 0 seconds 00010000:00010000:0.0:1788201607.873412:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201608.065704:0:29575:0:(client.c:2702:ptlrpc_set_wait()) set ffff902350db2a80 going to sleep for 0 seconds 00000080:00200000:0.0:1788201608.577239:0:11804:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x429:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201608.577254:0:11804:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201608.577307:0:11804:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x429:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1788201608.897049:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201609.089033:0:29575:0:(client.c:2702:ptlrpc_set_wait()) set ffff902350db2a80 going to sleep for 0 seconds 00010000:00010000:1.0:1788201609.925217:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201610.113790:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level REPLAY_LOCKS/6 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201610.113798:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY_LOCKS) 00000100:00100000:2.0:1788201610.114056:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442781824:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201610.114166:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201610.114174:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442781696:192.168.201.130@tcp:400:kworker.0 00000100:00000400:0.0:1788201610.114758:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201594/real 1788201594] req@ffff9023450b8380 x1875064442779520/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 328/400 e 0 to 1 dl 1788201610 ref 1 fl Rpc:XQr/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201610.114784:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY_LOCKS (now=REPLAY_LOCKS) req@ffff9023450b8380 x1875064442779520/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 328/400 e 0 to 1 dl 1788201610 ref 1 fl Rpc:XQU/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201610.114798:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to CONNECTING 00000100:00080000:0.0:1788201610.114804:0:2358:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 871 00000100:00080000:0.0:1788201610.114814:0:2358:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201610.114825:0:2358:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201610.114846:0:2358:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442781952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201610.114865:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442779520:192.168.201.130@tcp:101:ldlm_lock_repla.0 00000100:00100000:0.0:1788201610.114901:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442781952:192.168.201.130@tcp:38:ptlrpcd_rcv.0 00000100:00100000:2.0:1788201610.116108:0:29575:0:(client.c:2702:ptlrpc_set_wait()) set ffff902350db2a80 going to sleep for 0 seconds 00000100:00100000:3.0:1788201610.123993:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442781696:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201610.124016:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442781824:192.168.201.130@tcp:400:kworker.0 00000100:00080000:0.0:1788201610.138253:0:2358:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff902350a11000: connect to target with instance 6 00000100:00080000:0.0:1788201610.138284:0:2358:0:(import.c:1297:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@192.168.201.130@tcp after partition 00000100:00080000:0.0:1788201610.138287:0:2358:0:(import.c:1307:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff902350a11000: reconnected to lustre-MDT0000_UUID during replay 00000100:00080000:0.0:1788201610.138290:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:0.0:1788201610.138295:0:2358:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788201610.138298:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 21474836483 gen 2 00000100:00080000:0.0:1788201610.138302:0:2358:0:(recover.c:53:ptlrpc_replay_next()) import ffff902348897000 from lustre-MDT0000_UUID committed 21474836483 last 0 00000100:00080000:0.0:1788201610.138304:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:2.0:1788201610.140540:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442781952:192.168.201.130@tcp:38:ptlrpcd_rcv.0 00010000:00080000:0.0:1788201610.140635:0:29591:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff902350a11000 to lustre-MDT0000_UUID@192.168.201.130@tcp 00010000:00010000:2.0:1788201610.141378:0:29591:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d602a expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00080000:2.0:1788201610.141406:0:29591:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00010000:00010000:2.0:1788201610.141413:0:29591:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00080000:2.0:1788201610.141420:0:29591:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 2(8) 00000100:00100000:0.0:1788201610.142744:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442782080:192.168.201.130@tcp:101:ldlm_lock_repla.0 00000100:00100000:0.0:1788201610.142845:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442782208:192.168.201.130@tcp:101:ldlm_lock_repla.0 00010000:00010000:0.0:1788201610.154306:0:2358:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d620d expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1788201610.154335:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442782080:192.168.201.130@tcp:101:ldlm_lock_repla.0 00010000:00010000:0.0:1788201610.154351:0:2358:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000100:00080000:0.0:1788201610.154359:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:0.0:1788201610.154402:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442782208:192.168.201.130@tcp:101:ldlm_lock_repla.0 00000100:00100000:0.0:1788201610.154429:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442782336:192.168.201.130@tcp:400:ptlrpcd_rcv.0 00000400:00000100:3.0:1788201610.185919:0:2354:0:(lib-move.c:4357:lnet_parse_put()) Dropping PUT from 12345-192.168.201.130@tcp portal 10 match 1875064442779520 offset 224 length 224: 4 00010000:00010000:2.0:1788201610.945091:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201611.137212:0:29575:0:(client.c:2702:ptlrpc_set_wait()) set ffff902350db2a80 going to sleep for 0 seconds 00000100:00080000:0.0:1788201611.321753:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:0.0:1788201611.321779:0:2358:0:(client.c:3234:ptlrpc_resend_req()) @@@ going to resend req@ffff9023450b8380 x1875064442782336/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201626 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201611.321792:0:2358:0:(client.c:3243:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff9023450b8380 x1875064442782336/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201626 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201611.321799:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:0.0:1788201611.322603:0:2358:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff902350a11000: Connection restored to 192.168.201.130@tcp (at 192.168.201.130@tcp) 00000100:00080000:0.0:1788201611.322615:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000f6dd60bd): req@ffff9023450ba300 x1875064442781312/t0(0) o41->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201611.322688:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442782336:192.168.201.130@tcp:400:ptlrpcd_rcv.0 00000100:00100000:2.0:1788201611.322959:0:29575:0:(client.c:2702:ptlrpc_set_wait()) set ffff902350db2a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201611.337618:0:29575:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29575:1875064442781312:192.168.201.130@tcp:41:lfs.0 00800000:00000004:0.0:1788201611.337659:0:29575:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: update ffff902350be5190 cache blocks 17239/17271 objects 530860/531195 00000080:00200080:0.0:1788201611.337773:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337797:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337807:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337812:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337817:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337821:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337825:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337829:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337833:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337838:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337842:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337845:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337848:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337852:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337856:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337859:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337864:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337867:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337871:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337897:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337904:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337908:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337912:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337915:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337919:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337922:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337926:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337930:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337934:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337938:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337942:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337946:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337950:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337954:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337959:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337963:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337968:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337972:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337976:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337980:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337984:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337988:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.337992:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.337996:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338000:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338015:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338021:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338025:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338029:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338033:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338037:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338041:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338045:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338049:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338054:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338057:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338061:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338064:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338068:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338072:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338076:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338080:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338084:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338087:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338091:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338095:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338099:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338102:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338106:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338109:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338113:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338117:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338121:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338124:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338128:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338132:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338136:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338140:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338143:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338147:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338151:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338153:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338157:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338160:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338164:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338167:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338172:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338175:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338179:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338183:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338199:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338204:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338208:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338212:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338215:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338219:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338223:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338226:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338230:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338233:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338237:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338241:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338245:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338248:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338253:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338256:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338260:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338264:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338268:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338273:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338277:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338281:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338285:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338289:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338293:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338296:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338300:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338303:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338307:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338310:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338314:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338317:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338320:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338323:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338327:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338331:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338334:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00800000:00000080:0.0:1788201611.338338:0:29575:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00000080:00200080:0.0:1788201611.338341:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:0.0:1788201611.338348:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234587c000 uarg=0000000000000000 00020000:00000004:0.0:1788201611.338358:0:29575:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: age 718, max_age 886 00000100:00100000:0.0:1788201611.338402:0:29575:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29575:1875064442782464:192.168.201.130@tcp:13:lfs.0 00000100:00100000:0.0:1788201611.338512:0:29575:0:(client.c:2702:ptlrpc_set_wait()) set ffff902350db2a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201611.341675:0:29575:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29575:1875064442782464:192.168.201.130@tcp:13:lfs.0 00020000:00000004:1.0:1788201611.341712:0:29575:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: update ffff902350be1e10 cache blocks 3667/3683 objects 117728/118665 00000080:00200080:1.0:1788201611.341782:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.341801:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00020000:00000004:1.0:1788201611.341810:0:29575:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: age 719, max_age 886 00000100:00080000:1.0:1788201611.341825:0:29575:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 708 00000100:00080000:1.0:1788201611.341837:0:29575:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201611.341850:0:29575:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 3407872, lost: 0. 00000100:00100000:1.0:1788201611.341907:0:29575:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442782592/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201611.341956:0:29575:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff90234519ed80 x1875064442782720/t0(0) o13->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201611.341966:0:29575:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342e36cc0 going to sleep for 0 seconds 00000100:00100000:2.0:1788201611.342160:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442782592:192.168.201.130@tcp:8:lfs.0 00000100:00100000:0.0:1788201611.346655:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff902350a11000: committing for last_committed 4294968873 gen 4 00000100:00080000:0.0:1788201611.346680:0:2358:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff902350a11000: connect to target with instance 1 00000100:00080000:0.0:1788201611.346691:0:2358:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff902350a11000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:0.0:1788201611.346706:0:2358:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:0.0:1788201611.346712:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:0.0:1788201611.346724:0:2358:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1788201611.346730:0:2358:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:0.0:1788201611.346750:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000b43aa016): req@ffff90234519ed80 x1875064442782720/t0(0) o13->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201611.346781:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442782592:192.168.201.130@tcp:8:lfs.0 00000100:00100000:2.0:1788201611.347744:0:29575:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342e36cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201611.350725:0:29575:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29575:1875064442782720:192.168.201.130@tcp:13:lfs.0 00020000:00000004:1.0:1788201611.350765:0:29575:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: update ffff902351d65190 cache blocks 3679/3683 objects 117792/118185 00000080:00200080:1.0:1788201611.350871:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.350924:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.350933:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.350938:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.350942:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.350946:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.350950:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.350954:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.350958:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.350961:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.350965:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.350968:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.350972:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.350975:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.350978:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.350981:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.350984:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.350987:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.350993:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.350996:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.351000:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.351693:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.351708:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.351714:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.351718:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.351721:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.351725:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.351728:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.351731:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.351734:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.351738:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.351741:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.351744:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.351747:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.351751:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.351754:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.351757:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.351760:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.351764:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.351767:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.351770:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.351773:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.351777:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.352269:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.352279:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.352283:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.352287:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.352290:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.352294:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353093:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353103:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353107:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353111:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353114:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353118:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353121:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353124:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353127:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353131:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353134:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353137:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353140:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353143:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353146:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353150:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353153:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353156:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353159:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353163:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353166:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353170:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353172:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353176:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353179:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353182:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353583:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353594:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353599:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353603:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353606:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353609:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353612:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353616:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353619:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353623:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353626:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353630:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353633:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353638:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353641:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353645:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353648:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353652:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353656:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353660:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353663:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353667:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353670:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353674:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353677:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353680:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353683:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353687:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353689:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353692:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353695:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353698:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353701:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353705:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353708:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353713:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353716:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353720:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353723:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353726:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353730:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353733:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353736:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353740:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353744:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353748:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353751:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353755:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353758:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200080:1.0:1788201611.353763:0:29575:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe20b07be0 00020000:00000080:1.0:1788201611.353766:0:29575:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581fc00 uarg=0000000000000000 00000080:00200008:1.0:1788201611.356537:0:29575:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00000080:00200000:1.0:1788201611.429767:0:29189:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201611.429821:0:29189:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201611.429848:0:29189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201611.429867:0:29189:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201611.429902:0:29189:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201611.429915:0:29189:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201611.429919:0:29189:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c58e4 00000080:00010000:1.0:1788201611.429929:0:29189:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c58e4 from it ffffa0b180c87b58 00010000:00010000:1.0:1788201611.429934:0:29189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201611.429944:0:29189:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:1.0:1788201611.429973:0:29189:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201611.429984:0:29189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f121.replay-single, flags=9 00000080:00200008:1.0:1788201611.429997:0:29189:0:(file.c:6283:ll_getattr_dentry()) START file f121.replay-single:[0x200000bd1:0x429:0x0](ffff90234497d348), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201611.430015:0:29189:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x429:0x0](ffff90234497d348),name=f121.replay-single 00000002:00010000:1.0:1788201611.430028:0:29189:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x429:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201611.430036:0:29189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d620d expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201611.430045:0:29189:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d620d expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201611.430051:0:29189:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201611.430056:0:29189:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x429:0x0](ffff90234497d348) 00000080:00010000:1.0:1788201611.430058:0:29189:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x429:0x0](ffff90234497d348) for lock 0x4605c7c97e1c58eb 00000080:00010000:1.0:1788201611.430063:0:29189:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c58eb from it ffffa0b180c87cb8 00010000:00010000:1.0:1788201611.430066:0:29189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d620d expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201611.430073:0:29189:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d620d expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201611.430140:0:29189:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x429:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788201611.430153:0:29189:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x429:0x0] 00000008:00010000:1.0:1788201611.430193:0:29189:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788201611.430198:0:29189:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff90235266d130, osc lock ffff90235090d300, flags 0x400001002 00010000:00010000:1.0:1788201611.430221:0:29189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201611.430235:0:29189:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:1.0:1788201611.430250:0:29189:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1788201611.430254:0:29189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201611.430262:0:29189:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:1.0:1788201611.430275:0:29189:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1788201611.430288:0:29189:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x429:0x0] updating i_size 0 i_blocks 1 00000020:00010000:1.0:1788201611.430294:0:29189:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff90235844d030 (ffff9023500e71b8/0) at cl_lock_release():245 00000020:00010000:1.0:1788201611.430298:0:29189:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235844d030 (ffff9023500e71b8/0) at cl_lock_cancel():142 00000020:00010000:1.0:1788201611.430300:0:29189:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff90235266d130 (ffff9023500e7bd8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1788201611.430305:0:29189:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235844d030 (ffff9023500e71b8/0) at cl_lock_fini():72 00000020:00010000:1.0:1788201611.430307:0:29189:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff90235266d130 (ffff9023500e71b8/1) at cl_lock_fini():72 00000080:00200000:1.0:1788201611.430342:0:29189:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x429:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:0.0:1788201611.972873:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201612.993166:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201614.024537:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788201614.516227:0:29686:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201614.516268:0:29686:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201614.516291:0:29686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788201614.516314:0:29686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788201614.516324:0:29686:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201614.516334:0:29686:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201614.516339:0:29686:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c58e4 00000080:00010000:2.0:1788201614.516425:0:29686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c58e4 from it ffffa0b181c13b58 00010000:00010000:2.0:1788201614.516432:0:29686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788201614.516443:0:29686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:2.0:1788201614.516465:0:29686:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201614.516476:0:29686:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f121.replay-single, flags=8 00000080:00200008:2.0:1788201614.516488:0:29686:0:(file.c:6283:ll_getattr_dentry()) START file f121.replay-single:[0x200000bd1:0x429:0x0](ffff90234497d348), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201614.516494:0:29686:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x429:0x0](ffff90234497d348),name=f121.replay-single 00000002:00010000:2.0:1788201614.516509:0:29686:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x429:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201614.516519:0:29686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d620d expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201614.516528:0:29686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d620d expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201614.516535:0:29686:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201614.516540:0:29686:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x429:0x0](ffff90234497d348) 00000080:00010000:2.0:1788201614.516542:0:29686:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x429:0x0](ffff90234497d348) for lock 0x4605c7c97e1c58eb 00000080:00010000:2.0:1788201614.516546:0:29686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c58eb from it ffffa0b181c13cb8 00010000:00010000:2.0:1788201614.516549:0:29686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d620d expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201614.516557:0:29686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d620d expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201614.516622:0:29686:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x429:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1788201614.516634:0:29686:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000bd1:0x429:0x0] 00000008:00010000:2.0:1788201614.516677:0:29686:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1788201614.516681:0:29686:0:(osc_lock.c:1310:osc_lock_init()) ### lock ffff9023533980b0, osc lock ffff9023500dc4c0, flags 0x400001002 00010000:00010000:2.0:1788201614.517088:0:29686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201614.517106:0:29686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 2/1,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:2.0:1788201614.517122:0:29686:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1788201614.517127:0:29686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 5/1,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201614.517137:0:29686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 4/0,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000020:00200000:2.0:1788201614.517152:0:29686:0:(cl_io.c:1872:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1788201614.517166:0:29686:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000bd1:0x429:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1788201614.517170:0:29686:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff902350cc6e40 (ffff902350b9d248/0) at cl_lock_release():245 00000020:00010000:2.0:1788201614.517174:0:29686:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff902350cc6e40 (ffff902350b9d248/0) at cl_lock_cancel():142 00000020:00010000:2.0:1788201614.517177:0:29686:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9023533980b0 (ffff902350b9d5a8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1788201614.517182:0:29686:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff902350cc6e40 (ffff902350b9d248/0) at cl_lock_fini():72 00000020:00010000:2.0:1788201614.517185:0:29686:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9023533980b0 (ffff902350b9d248/1) at cl_lock_fini():72 00000080:00200000:2.0:1788201614.517221:0:29686:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x429:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201614.517265:0:29686:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201614.517275:0:29686:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201614.517283:0:29686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788201614.517292:0:29686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:2.0:1788201614.517299:0:29686:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201614.517304:0:29686:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201614.517307:0:29686:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c58e4 00000080:00010000:2.0:1788201614.517312:0:29686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c58e4 from it ffffa0b181c13c10 00010000:00010000:2.0:1788201614.517315:0:29686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788201614.517323:0:29686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788201614.517332:0:29686:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201614.517341:0:29686:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f121.replay-single, flags=0 00000080:00200000:2.0:1788201614.519281:0:29686:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201614.519323:0:29686:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201614.519405:0:29686:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788201614.519427:0:29686:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788201614.519437:0:29686:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201614.519449:0:29686:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201614.519452:0:29686:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c58e4 00000080:00010000:2.0:1788201614.519461:0:29686:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c58e4 from it ffffa0b181c13d68 00010000:00010000:2.0:1788201614.519465:0:29686:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788201614.519474:0:29686:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788201614.519491:0:29686:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 3 00000080:00200000:2.0:1788201614.519501:0:29686:0:(namei.c:2418:ll_unlink()) VFS Op:name=f121.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908) 00010000:00010000:2.0:1788201614.519591:0:29686:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788201614.519625:0:29686:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d620d expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201614.519646:0:29686:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x429:0x0]: losing layout lock 00010000:00010000:2.0:1788201614.519700:0:29686:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d620d expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201614.519709:0:29686:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788201614.519715:0:29686:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201614.519719:0:29686:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865b400/0x4605c7c97e1c58eb lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x429:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d620d expref: -99 pid: 29189 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201614.519948:0:29686:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684c00/0x4605c7c97e1c58e4 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d61f8 expref: -99 pid: 29189 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000020:00100000:2.0:1788201614.519977:0:29686:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201614.519993:0:29686:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234598c000 x1875064442782976/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201614.520024:0:29686:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c000 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29686:1875064442782976:192.168.201.130@tcp:36:rm.0 00000100:00100000:2.0:1788201614.520153:0:29686:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234502f600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201614.628116:0:29686:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c000 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:29686:1875064442782976:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201614.628248:0:29686:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x429:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201614.628263:0:29686:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201614.628329:0:29686:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x429:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201614.628363:0:29686:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x429:0x0](ffff90234497d348) 00000008:00010000:0.0:1788201614.628422:0:29686:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 1 [cma]time: 1788201577 1788201577 1788201577 by oinfo size 0 blocks 1 [cma]time 1788201577 1788201577 1788201577 ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 1/0,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201615.048092:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201615.235966:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442783104:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201615.236099:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442783360:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201615.236141:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201615.236147:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442783232:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201615.239608:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442783232:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201615.239649:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442783360:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201615.241716:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442783104:192.168.201.130@tcp:400:kworker.0 00010000:00010000:1.0:1788201616.072177:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201616.681547:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201616.681598:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458332800 00000100:00100000:3.0:1788201616.681611:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458332800 00000100:00100000:3.0:1788201616.681678:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1368 00000100:00100000:3.0:1788201616.681690:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902352689c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13447:x1875064458332800:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201616.681715:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201616.681814:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201616.681821:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902352689c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:13447:x1875064458332800:12345-192.168.201.130@tcp:104: Request processed in 134us (291us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201616.681828:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1368 00010000:00010000:1.0:1788201616.682033:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201616.682050:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201616.682067:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865b200 already unused, calling callback (ffffffffc11352d0) 00010000:00010000:1.0:1788201616.682074:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 3/0,0 mode: PR/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201616.682192:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 2/0,0 mode: --/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201616.682202:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201616.682229:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201616.682237:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e196bb69/0x4605c7c97e1c58f2 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x482:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x9383f985bc4d6031 expref: -99 pid: 29189 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201616.682264:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201616.682266:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201616.682267:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201616.682277:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201616.682294:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442783488:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201616.690198:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442783488:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201617.103784:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201618.118655:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201619.152478:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201620.170935:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201620.354988:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442783616:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201620.355093:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442783872:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201620.355127:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201620.355134:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442783744:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201620.355156:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442784000:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201620.361755:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442783616:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201620.361777:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442783872:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201620.371576:0:2362:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 25769803777 gen 2 00000100:00100000:3.0:1788201620.371605:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442783744:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201620.372241:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442784000:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201621.186278:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201622.211099:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201623.236036:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201623.942703:0:30244:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 130a: DoM file create (setstripe) replay ========================================================== 14:40:22 (1788201622) 00010000:00010000:1.0:1788201624.263128:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788201625.168586:0:30268:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201625.168601:0:30268:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201625.168623:0:30268:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201625.168691:0:30268:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 30268 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201625.168701:0:30268:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 30268 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201625.168712:0:30268:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 30268 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201625.168737:0:30268:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30268:1875064442784128:192.168.201.130@tcp:101:lfs.0 00000100:00100000:3.0:1788201625.168926:0:30268:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b240 going to sleep for 11 seconds 00000100:00100000:3.0:1788201625.178613:0:30268:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30268:1875064442784128:192.168.201.130@tcp:101:lfs.0 00010000:00010000:3.0:1788201625.178642:0:30268:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201625.178660:0:30268:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201625.178670:0:30268:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201625.178680:0:30268:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201625.178693:0:30268:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201625.178701:0:30268:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201625.178709:0:30268:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201625.178721:0:30268:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352689880 x1875064442784128/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201641 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201625.178735:0:30268:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201625.178747:0:30268:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201625.178754:0:30268:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201625.178785:0:30268:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:3.0:1788201625.178792:0:30268:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5915 00000080:00010000:3.0:1788201625.178797:0:30268:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201625.178800:0:30268:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5915 from it ffffa0b180c87cb8 00010000:00010000:3.0:1788201625.178804:0:30268:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201625.178813:0:30268:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788201625.181377:0:30268:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788201625.181388:0:30268:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201625.181410:0:30268:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201625.181428:0:30268:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201625.181443:0:30268:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201625.181451:0:30268:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201625.181458:0:30268:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201625.181462:0:30268:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5915 00000080:00010000:3.0:1788201625.181468:0:30268:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5915 from it ffffa0b180c87cb8 00010000:00010000:3.0:1788201625.181472:0:30268:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201625.181481:0:30268:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201625.181614:0:30268:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201625.181623:0:30268:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201625.181630:0:30268:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201625.181640:0:30268:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201625.181646:0:30268:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201625.181650:0:30268:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201625.181652:0:30268:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5915 00000080:00010000:3.0:1788201625.181656:0:30268:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5915 from it ffffa0b180c87bb0 00010000:00010000:3.0:1788201625.181659:0:30268:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201625.181666:0:30268:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201625.181674:0:30268:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 44 00000080:00200008:3.0:1788201625.181681:0:30268:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00000080:00200080:3.0:1788201625.181844:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.181859:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00800000:00000004:3.0:1788201625.181866:0:30268:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: age 887, max_age 900 00000100:00100000:3.0:1788201625.181930:0:30268:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30268:1875064442784256:192.168.201.130@tcp:41:lfs.0 00000100:00100000:3.0:1788201625.182467:0:30268:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b240 going to sleep for 11 seconds 00000100:00100000:3.0:1788201625.186550:0:30268:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30268:1875064442784256:192.168.201.130@tcp:41:lfs.0 00800000:00000004:3.0:1788201625.186582:0:30268:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: update ffff902350be5190 cache blocks 17239/17271 objects 530726/531060 00000080:00200080:3.0:1788201625.186694:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186710:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186717:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186721:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186725:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186729:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186733:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186737:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186741:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186744:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186748:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186751:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186755:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186758:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186762:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186765:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186769:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186772:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186777:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186780:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186784:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186788:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186792:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186796:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186800:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186803:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186807:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186810:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186814:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186817:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186821:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186825:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186829:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186833:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186837:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186841:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186845:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186848:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186852:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186856:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186860:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186863:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186867:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186870:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186873:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186909:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186915:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186918:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186922:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186927:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186930:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186934:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186937:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186941:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186953:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186958:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186963:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186966:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186970:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186973:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186977:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186981:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186984:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186988:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.186991:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.186995:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187000:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187003:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187007:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187010:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187014:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187017:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187021:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187025:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187029:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187033:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187036:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187039:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187043:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187046:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187049:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187053:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187056:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187060:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187063:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187066:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187070:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187073:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187077:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187080:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187085:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187088:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187092:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187095:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187099:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187103:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187107:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187110:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187114:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187117:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187121:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187124:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187127:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187131:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187134:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187460:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187470:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187474:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187478:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187481:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187484:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187487:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187491:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187495:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187499:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187502:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187506:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187509:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187513:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187516:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187519:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187522:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187526:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187529:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187534:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187537:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187541:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00800000:00000080:3.0:1788201625.187545:0:30268:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00000080:00200080:3.0:1788201625.187550:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:3.0:1788201625.187556:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff9023584a5400 uarg=0000000000000000 00020000:00000004:3.0:1788201625.187565:0:30268:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: age 887, max_age 900 00000100:00100000:3.0:1788201625.187603:0:30268:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30268:1875064442784384:192.168.201.130@tcp:13:lfs.0 00000100:00100000:3.0:1788201625.188446:0:30268:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b240 going to sleep for 11 seconds 00000100:00100000:0.0:1788201625.189706:0:30268:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30268:1875064442784384:192.168.201.130@tcp:13:lfs.0 00020000:00000004:0.0:1788201625.189734:0:30268:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: update ffff902350be1e10 cache blocks 3667/3683 objects 117728/118665 00000080:00200080:0.0:1788201625.193426:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.193456:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00020000:00000004:0.0:1788201625.193466:0:30268:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: age 887, max_age 900 00000100:00100000:0.0:1788201625.193517:0:30268:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30268:1875064442784512:192.168.201.130@tcp:13:lfs.0 00000100:00100000:0.0:1788201625.193841:0:30268:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201625.197591:0:30268:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30268:1875064442784512:192.168.201.130@tcp:13:lfs.0 00020000:00000004:0.0:1788201625.197619:0:30268:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: update ffff902351d65190 cache blocks 3679/3683 objects 117792/118185 00000080:00200080:0.0:1788201625.197686:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197703:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197709:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197723:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197727:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197732:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197736:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197739:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197743:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197746:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197749:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197756:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197761:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197764:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197768:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197771:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197776:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197779:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197783:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197786:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197789:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197793:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197796:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197800:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197803:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197807:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197810:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197813:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197817:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197819:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197823:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197826:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197830:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197833:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197837:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197841:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197844:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197847:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197851:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197854:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197858:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197862:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197866:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197869:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197873:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197896:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197901:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197905:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197908:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197911:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197915:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197918:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197922:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197925:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197929:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197932:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197936:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197939:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197943:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197959:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197964:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197967:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197971:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197974:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197978:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197981:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197986:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197989:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.197993:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.197996:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198000:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198003:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198007:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198010:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198014:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198017:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198021:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198024:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198028:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198031:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198035:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198038:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198042:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198045:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198049:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198053:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198067:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198071:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198075:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198078:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198081:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198093:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198097:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198100:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198104:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198107:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198110:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198114:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198117:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198120:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198124:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198127:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198130:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198133:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198146:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198151:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198155:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198158:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198162:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198165:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198169:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198180:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198185:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198189:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198193:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198196:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198200:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198203:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198207:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198211:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198215:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198218:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198221:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198224:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201625.198228:0:30268:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffe4fb4c5f0 00020000:00000080:0.0:1788201625.198231:0:30268:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200008:0.0:1788201625.198245:0:30268:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00000080:00200000:3.0:1788201625.257871:0:30275:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201625.257928:0:30275:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201625.257948:0:30275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201625.257965:0:30275:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201625.257973:0:30275:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201625.257981:0:30275:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201625.257984:0:30275:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5915 00000080:00010000:3.0:1788201625.257992:0:30275:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5915 from it ffffa0b181bfbbd0 00010000:00010000:3.0:1788201625.257997:0:30275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201625.258006:0:30275:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788201625.258028:0:30275:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201625.258041:0:30275:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg130-client.virtnet, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=1536 00000080:00200000:3.0:1788201625.258045:0:30275:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201625.258051:0:30275:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201625.258057:0:30275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201625.258064:0:30275:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788201625.258070:0:30275:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201625.258073:0:30275:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201625.258076:0:30275:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5915 00000080:00010000:3.0:1788201625.258079:0:30275:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5915 from it ffffa0b181bfbcd0 00010000:00010000:3.0:1788201625.258082:0:30275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788201625.258088:0:30275:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201625.258096:0:30275:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 3 00000080:00200000:3.0:1788201625.258102:0:30275:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201625.258107:0:30275:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201625.258113:0:30275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201625.258125:0:30275:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788201625.258131:0:30275:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201625.258135:0:30275:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201625.258138:0:30275:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5915 00000080:00010000:3.0:1788201625.258141:0:30275:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5915 from it ffffa0b181bfbdb0 00010000:00010000:3.0:1788201625.258144:0:30275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788201625.258151:0:30275:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788201625.258158:0:30275:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 3 00000080:00200000:3.0:1788201625.258163:0:30275:0:(namei.c:2129:ll_create_nd()) VFS Op:name=fsa-oleg130-client.virtnet, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=33188, excl=1 00000080:00200000:3.0:1788201625.258167:0:30275:0:(namei.c:2084:ll_mknod()) VFS Op:name=fsa-oleg130-client.virtnet, dir=[0x200000007:0x1:0x0](ffff902352108908) mode 100644 dev 0 00010000:00010000:3.0:1788201625.258204:0:30275:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788201625.258254:0:30275:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788201625.258261:0:30275:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201625.258264:0:30275:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685800/0x4605c7c97e1c5915 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d63fe expref: -99 pid: 30268 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000020:00100000:3.0:1788201625.258288:0:30275:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201625.258300:0:30275:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff902352688380 x1875064442784768/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201625.258325:0:30275:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job mcreate:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30275:1875064442784768:192.168.201.130@tcp:36:mcreate.0 00000100:00100000:3.0:1788201625.258422:0:30275:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b780 going to sleep for 11 seconds 00000100:00100000:0.0:1788201625.276725:0:30275:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job mcreate:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30275:1875064442784768:192.168.201.130@tcp:36:mcreate.0 00000080:00200000:0.0:1788201625.276785:0:30275:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x42a:0x0](ffff9023449b6c08) 00000080:00200000:0.0:1788201625.276870:0:30275:0:(namei.c:2137:ll_create_nd()) VFS Op:name=fsa-oleg130-client.virtnet, unhashed 1 00010000:00010000:2.0:1788201625.283245:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1788201625.296918:0:30278:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201625.296948:0:30278:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201625.297087:0:30278:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 30278 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.297100:0:30278:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 30278 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.297113:0:30278:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 30278 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201625.297152:0:30278:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30278:1875064442784896:192.168.201.130@tcp:101:rm.0 00000100:00100000:0.0:1788201625.297257:0:30278:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201625.301280:0:30278:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30278:1875064442784896:192.168.201.130@tcp:101:rm.0 00010000:00010000:0.0:1788201625.301307:0:30278:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.301321:0:30278:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.301334:0:30278:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.301345:0:30278:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.301359:0:30278:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.301840:0:30278:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.301848:0:30278:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201625.301861:0:30278:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023450b8380 x1875064442784896/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201641 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201625.301913:0:30278:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.301926:0:30278:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201625.301935:0:30278:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201625.302227:0:30278:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:0.0:1788201625.302235:0:30278:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c591c 00000080:00010000:0.0:1788201625.302241:0:30278:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201625.302244:0:30278:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c591c from it ffffa0b181bfbb58 00010000:00010000:0.0:1788201625.302248:0:30278:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.302257:0:30278:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201625.302282:0:30278:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201625.302304:0:30278:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg130-client.virtnet, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=8 00000080:00200000:0.0:1788201625.302308:0:30278:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg130-client.virtnet, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=getattr 00000002:00010000:0.0:1788201625.302328:0:30278:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: fsa-oleg130-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201625.302367:0:30278:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 30278 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.302374:0:30278:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 30278 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.302384:0:30278:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 30278 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201625.302407:0:30278:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30278:1875064442785024:192.168.201.130@tcp:101:rm.0 00000100:00100000:0.0:1788201625.302503:0:30278:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201625.306433:0:30278:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30278:1875064442785024:192.168.201.130@tcp:101:rm.0 00010000:00010000:0.0:1788201625.306462:0:30278:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.307595:0:30278:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.307616:0:30278:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.307626:0:30278:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.307639:0:30278:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.307655:0:30278:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.307664:0:30278:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.307672:0:30278:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201625.307685:0:30278:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9023450b8380 x1875064442785024/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 600/584 e 0 to 0 dl 1788201641 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201625.307699:0:30278:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.307715:0:30278:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1066 (0 0) 00000080:00200000:0.0:1788201625.307723:0:30278:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201625.307752:0:30278:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x42a:0x0], updating i_size 0 00000080:00200000:0.0:1788201625.307756:0:30278:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x42a:0x0](ffff9023449b6c08): rc = 0 00000080:00200000:0.0:1788201625.307802:0:30278:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x42a:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:0.0:1788201625.307824:0:30278:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42a:0x0](ffff9023449b6c08) for lock 0x4605c7c97e1c5923 00000080:00010000:0.0:1788201625.307835:0:30278:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42a:0x0](ffff9023449b6c08) 00000080:00010000:0.0:1788201625.307839:0:30278:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5923 from it ffffa0b181bfbb20 00010000:00010000:0.0:1788201625.307842:0:30278:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201625.307850:0:30278:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201625.307903:0:30278:0:(file.c:6283:ll_getattr_dentry()) START file fsa-oleg130-client.virtnet:[0x200000bd1:0x42a:0x0](ffff9023449b6c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201625.307911:0:30278:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x42a:0x0](ffff9023449b6c08),name=fsa-oleg130-client.virtnet 00000002:00010000:0.0:1788201625.307933:0:30278:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x42a:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201625.308071:0:30278:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201625.308084:0:30278:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201625.308091:0:30278:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201625.308096:0:30278:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42a:0x0](ffff9023449b6c08) 00000080:00010000:0.0:1788201625.308100:0:30278:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42a:0x0](ffff9023449b6c08) for lock 0x4605c7c97e1c5923 00000080:00010000:0.0:1788201625.308104:0:30278:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5923 from it ffffa0b181bfbcb8 00010000:00010000:0.0:1788201625.308108:0:30278:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201625.308116:0:30278:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201625.308185:0:30278:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201625.308193:0:30278:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201625.308201:0:30278:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201625.308209:0:30278:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201625.308215:0:30278:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201625.308219:0:30278:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201625.308221:0:30278:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c591c 00000080:00010000:0.0:1788201625.308225:0:30278:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c591c from it ffffa0b181bfbc10 00010000:00010000:0.0:1788201625.308227:0:30278:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201625.308234:0:30278:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201625.308242:0:30278:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201625.308251:0:30278:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg130-client.virtnet, flags=0 00000080:00200000:0.0:1788201625.308255:0:30278:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201625.308261:0:30278:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201625.308267:0:30278:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201625.308275:0:30278:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201625.308281:0:30278:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201625.308285:0:30278:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201625.308287:0:30278:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c591c 00000080:00010000:0.0:1788201625.308290:0:30278:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c591c from it ffffa0b181bfbd68 00010000:00010000:0.0:1788201625.308293:0:30278:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201625.308300:0:30278:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201625.308308:0:30278:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 3 00000080:00200000:0.0:1788201625.308314:0:30278:0:(namei.c:2418:ll_unlink()) VFS Op:name=fsa-oleg130-client.virtnet, dir=[0x200000007:0x1:0x0](ffff902352108908) 00010000:00010000:0.0:1788201625.308349:0:30278:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201625.308372:0:30278:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201625.308389:0:30278:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x42a:0x0]: losing layout lock 00010000:00010000:0.0:1788201625.308427:0:30278:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x42a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201625.308439:0:30278:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201625.308444:0:30278:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201625.308447:0:30278:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865a400/0x4605c7c97e1c5923 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x42a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d6436 expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201625.308465:0:30278:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865ba00/0x4605c7c97e1c591c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d641a expref: -99 pid: 30278 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1788201625.308484:0:30278:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201625.309610:0:30278:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450b8380 x1875064442785280/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201625.309642:0:30278:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30278:1875064442785280:192.168.201.130@tcp:36:rm.0 00000100:00100000:0.0:1788201625.309863:0:30278:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201625.322185:0:30278:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30278:1875064442785280:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201625.322287:0:30278:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x42a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788201625.322297:0:30278:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x42a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201625.322320:0:30278:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x42a:0x0](ffff9023449b6c08) 00000100:00100000:0.0:1788201625.473361:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442785536:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201625.473481:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442785792:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201625.473515:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1788201625.473521:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442785408:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201625.473551:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442785664:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201625.479462:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442785408:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201625.479547:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442785664:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201625.479563:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442785536:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201625.479575:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442785792:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201626.306136:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201627.329403:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201628.354833:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201628.474635:0:30308:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00000080:00200000:2.0:1788201628.495087:0:30309:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201628.495136:0:30309:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201628.495219:0:30309:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.495231:0:30309:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.495242:0:30309:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201628.495273:0:30309:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ea00 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30309:1875064442785920:192.168.201.130@tcp:101:lfs.0 00000100:00100000:2.0:1788201628.495628:0:30309:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351938540 going to sleep for 11 seconds 00000100:00100000:2.0:1788201628.499964:0:30309:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ea00 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30309:1875064442785920:192.168.201.130@tcp:101:lfs.0 00010000:00010000:2.0:1788201628.500014:0:30309:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.500032:0:30309:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.500047:0:30309:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.500061:0:30309:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.500079:0:30309:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.500092:0:30309:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.500108:0:30309:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201628.500132:0:30309:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff90234598ea00 x1875064442785920/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201644 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201628.500150:0:30309:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.500167:0:30309:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201628.500177:0:30309:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201628.500218:0:30309:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:2.0:1788201628.500225:0:30309:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c592a 00000080:00010000:2.0:1788201628.500231:0:30309:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201628.500235:0:30309:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c592a from it ffffa0b181c33b58 00010000:00010000:2.0:1788201628.500238:0:30309:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.500247:0:30309:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201628.500277:0:30309:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201628.500307:0:30309:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f130a.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=8 00000080:00200000:2.0:1788201628.500312:0:30309:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f130a.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=getattr 00000002:00010000:2.0:1788201628.500338:0:30309:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f130a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201628.500383:0:30309:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c5931 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.500390:0:30309:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c5931 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.500399:0:30309:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c5931 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201628.500419:0:30309:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ea00 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30309:1875064442786048:192.168.201.130@tcp:101:lfs.0 00000100:00100000:2.0:1788201628.500505:0:30309:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351938540 going to sleep for 11 seconds 00000100:00100000:2.0:1788201628.508790:0:30309:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ea00 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30309:1875064442786048:192.168.201.130@tcp:101:lfs.0 00010000:00010000:2.0:1788201628.508815:0:30309:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c5931 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.508829:0:30309:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c5931 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.508836:0:30309:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c5931 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.508842:0:30309:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c5931 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.508850:0:30309:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c5931 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.508857:0:30309:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345106200 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:2.0:1788201628.508862:0:30309:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c5931 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.508869:0:30309:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c5931 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.509163:0:30309:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c5931 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.509172:0:30309:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201628.509181:0:30309:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c5931 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201628.509189:0:30309:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106200/0x4605c7c97e1c5931 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201628.509210:0:30309:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff90234598ea00 x1875064442786048/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201644 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201628.510314:0:30309:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1788201628.513787:0:30309:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201628.513812:0:30309:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788201628.513823:0:30309:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c592a from it ffffa0b181c33930 00010000:00010000:1.0:1788201628.513829:0:30309:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201628.513839:0:30309:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201628.513966:0:30309:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201628.513986:0:30309:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201628.513993:0:30309:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201628.514012:0:30309:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201628.514019:0:30309:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201628.514026:0:30309:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201628.514030:0:30309:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c592a 00000080:00010000:1.0:1788201628.514036:0:30309:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c592a from it ffffa0b181c33bf8 00010000:00010000:1.0:1788201628.514039:0:30309:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201628.514046:0:30309:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201628.514054:0:30309:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201628.514068:0:30309:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f130a.replay-single, flags=777 00000080:00200000:1.0:1788201628.514074:0:30309:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201628.514079:0:30309:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201628.514084:0:30309:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201628.514093:0:30309:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201628.514099:0:30309:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201628.514104:0:30309:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201628.514106:0:30309:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c592a 00000080:00010000:1.0:1788201628.514109:0:30309:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c592a from it ffffa0b181c33b38 00010000:00010000:1.0:1788201628.514112:0:30309:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201628.514127:0:30309:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201628.514136:0:30309:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 3 00000080:00200000:1.0:1788201628.514144:0:30309:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f130a.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), file ffff902351dbf080, open_flags a141, mode 81b6 opened 0 00000080:00200000:1.0:1788201628.514155:0:30309:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f130a.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=open|creat 00000002:00010000:1.0:1788201628.514176:0:30309:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f130a.replay-single,[0x200000bd1:0x42b:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0120502 00010000:00010000:1.0:1788201628.514188:0:30309:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201628.514368:0:30309:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201628.514377:0:30309:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201628.514379:0:30309:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345107000/0x4605c7c97e1c592a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d6460 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201628.514422:0:30309:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201628.514429:0:30309:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201628.514442:0:30309:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201628.514455:0:30309:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234519c700 x1875064442786304/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 536/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201628.514468:0:30309:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201628.514492:0:30309:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30309:1875064442786304:192.168.201.130@tcp:101:lfs.0 00000100:00100000:1.0:1788201628.514673:0:30309:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234738db40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201628.520444:0:30309:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30309:1875064442786304:192.168.201.130@tcp:101:lfs.0 00010000:00010000:1.0:1788201628.520478:0:30309:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201628.520514:0:30309:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201628.520524:0:30309:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 4/0,1 mode: --/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201628.520536:0:30309:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201628.520548:0:30309:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 4/0,1 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201628.520566:0:30309:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201628.520574:0:30309:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 5/1,1 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201628.520581:0:30309:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201628.520593:0:30309:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff90234519c700 x1875064442786304/t25769803780(25769803780) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201644 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201628.520607:0:30309:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90234519c700 x1875064442786304/t25769803780(25769803780) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201644 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201628.520620:0:30309:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201628.520630:0:30309:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1067 (0 0) 00000080:00200000:1.0:1788201628.520638:0:30309:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201628.520654:0:30309:0:(namei.c:1628:ll_create_it()) VFS Op:name=f130a.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=open|creat 00000080:00200000:1.0:1788201628.520684:0:30309:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x42b:0x0](ffff9023448e21c8) 00000080:00200000:1.0:1788201628.520703:0:30309:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x42b:0x0], updating i_size 0 00000080:00200000:1.0:1788201628.520794:0:30309:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x42b:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:1.0:1788201628.520804:0:30309:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x42b:0x0](ffff9023448e21c8) 00000080:00010000:1.0:1788201628.520807:0:30309:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42b:0x0](ffff9023448e21c8) for lock 0x4605c7c97e1c5938 00000080:00010000:1.0:1788201628.520817:0:30309:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42b:0x0](ffff9023448e21c8) 00000080:00200008:1.0:1788201628.520830:0:30309:0:(file.c:1090:ll_file_open()) START file f130a.replay-single:[0x200000bd1:0x42b:0x0](ffff9023448e21c8), flags 120501 00000002:00100000:1.0:1788201628.520849:0:30309:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90234519c700 x1875064442786304/t25769803780(25769803780) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201644 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201628.520916:0:30309:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5938 from it ffff9023450e7480 00010000:00010000:1.0:1788201628.520921:0:30309:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201628.520930:0:30309:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201628.520951:0:30309:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x42b:0x0](ffff9023448e21c8), inode mode 81a4 mask 40 00000080:00200000:1.0:1788201628.521009:0:30309:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x42b:0x0](ffff9023448e21c8), inode mode 81a4 mask 2 00000080:00200000:1.0:1788201628.521020:0:30309:0:(xattr.c:413:ll_xattr_set()) VFS Op:inode=[0x200000bd1:0x42b:0x0](ffff9023448e21c8), xattr lov 00000002:00010000:1.0:1788201628.521038:0:30309:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x42b:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 040000002 00010000:00010000:1.0:1788201628.521085:0:30309:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c593f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201628.521093:0:30309:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c593f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201628.521104:0:30309:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201628.521111:0:30309:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234519df80 x1875064442786560/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 712/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201628.521132:0:30309:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c593f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201628.521153:0:30309:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30309:1875064442786560:192.168.201.130@tcp:101:lfs.0 00000100:00100000:1.0:1788201628.521756:0:30309:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234738d6c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201628.526749:0:2353:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201628.526917:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458335744 00000100:00100000:3.0:1788201628.526927:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458335744 00000100:00100000:3.0:1788201628.526990:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1369 00000100:00100000:3.0:1788201628.527002:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90234598d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:35604eb4-baf9-45be-a20a-4883a2cd8cf9+5:23366:x1875064458335744:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201628.527022:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201628.527112:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201628.527127:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90234598d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:35604eb4-baf9-45be-a20a-4883a2cd8cf9+5:23366:x1875064458335744:12345-192.168.201.130@tcp:104: Request processed in 129us (394us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201628.527134:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1369 00010000:00010000:1.0:1788201628.527552:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201628.527560:0:13084:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201628.527570:0:13084:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265ce00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201628.527576:0:13084:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201628.527585:0:13084:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201628.527594:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 3/0,0 mode: PR/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201628.527650:0:13084:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x42b:0x0]: losing layout lock 00010000:00010000:1.0:1788201628.527771:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 2/0,0 mode: --/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201628.527780:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201628.527803:0:13084:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201628.527811:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c5938 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x42b:0x0].0x0 bits 0x1b/0x1b rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x9383f985bc4d6491 expref: -99 pid: 30309 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201628.527834:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201628.527836:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201628.527838:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201628.527847:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201628.527864:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442786688:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201628.534779:0:30309:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30309:1875064442786560:192.168.201.130@tcp:101:lfs.0 00010000:00010000:0.0:1788201628.534805:0:30309:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c593f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201628.534818:0:30309:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c593f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201628.534825:0:30309:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c593f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201628.534832:0:30309:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c593f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201628.534840:0:30309:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c593f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201628.534847:0:30309:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265d400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201628.534851:0:30309:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c593f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201628.534859:0:30309:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c593f lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201628.534866:0:30309:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c593f lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201628.534909:0:30309:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201628.534917:0:30309:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c593f lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201628.534924:0:30309:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d400/0x4605c7c97e1c593f lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 30309 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201628.534957:0:30309:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x800002b, status=0 req@ffff90234519df80 x1875064442786560/t25769803781(25769803781) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 712/824 e 0 to 0 dl 1788201644 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201628.534971:0:30309:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90234519df80 x1875064442786560/t25769803781(25769803781) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 712/824 e 0 to 0 dl 1788201644 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201628.534991:0:30309:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201628.535108:0:30309:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x42b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788201628.535124:0:30309:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x42b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:0.0:1788201628.535193:0:30309:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30309:1875064442786816:192.168.201.130@tcp:900:lfs.0 00000100:00100000:0.0:1788201628.535277:0:30309:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201628.536106:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442786688:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201628.550865:0:30309:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30309:1875064442786816:192.168.201.130@tcp:900:lfs.0 00000080:00200000:0.0:1788201628.550988:0:30309:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x42b:0x0], updating i_size 0 00000002:00100000:0.0:1788201628.551026:0:30309:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90234519df80 x1875064442786560/t25769803781(25769803781) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 736/824 e 0 to 0 dl 1788201644 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201628.551063:0:30309:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff90234519df80 x1875064442786560/t25769803781(25769803781) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 736/824 e 0 to 0 dl 1788201644 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201628.551072:0:30309:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201628.551076:0:30309:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450b9c00 x1875064442787072/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201628.551095:0:30309:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30309:1875064442787072:192.168.201.130@tcp:35:lfs.0 00000100:00100000:0.0:1788201628.551183:0:30309:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201628.554865:0:30309:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30309:1875064442787072:192.168.201.130@tcp:35:lfs.0 00000002:00100000:0.0:1788201628.554926:0:30309:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff90234519df80 x1875064442786560/t25769803781(25769803781) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 736/824 e 0 to 0 dl 1788201644 ref 2 fl Complete:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788201628.554984:0:30309:0:(file.c:479:ll_file_release()) START file f130a.replay-single:[0x200000bd1:0x42b:0x0](ffff9023448e21c8), flags 100001 00000002:00100000:0.0:1788201628.555110:0:30309:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff90234519c700 x1875064442786304/t25769803780(25769803780) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201644 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201628.555131:0:30309:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201628.555137:0:30309:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450b8700 x1875064442787328/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201628.555156:0:30309:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30309:1875064442787328:192.168.201.130@tcp:35:lfs.0 00000100:00100000:0.0:1788201628.555582:0:30309:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201628.557241:0:30309:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30309:1875064442787328:192.168.201.130@tcp:35:lfs.0 00000002:00100000:0.0:1788201628.557266:0:30309:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff90234519c700 x1875064442786304/t25769803780(25769803780) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201644 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788201629.377457:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201630.401698:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201630.594060:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442787584:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201630.594226:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442787840:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201630.594262:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:0.0:1788201630.594269:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442787456:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201630.594294:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442787712:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201630.610107:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442787584:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201630.610129:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442787840:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201630.610157:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442787456:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201630.610169:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442787712:192.168.201.130@tcp:400:kworker.0 00010000:00010000:1.0:1788201631.425446:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201632.449263:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201633.473108:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201634.502133:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201635.521327:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201635.713188:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442787968:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201635.713300:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442788224:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201635.713334:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1788201635.713339:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442788096:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201635.713358:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442788352:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201635.723517:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442788352:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201635.723540:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442788224:192.168.201.130@tcp:400:kworker.0 00010000:00010000:1.0:1788201636.545236:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201637.569226:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201638.593109:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201639.617037:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201640.641276:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201640.834186:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442788480:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201640.845781:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201640.845807:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442788608:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201640.850085:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442788864:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201640.850184:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201640.850190:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442788736:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201640.855404:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442788736:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201640.855422:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442788864:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201641.665248:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201642.689154:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201643.714325:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201644.737821:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201645.763393:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201645.953393:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:02000000:3.0:1788201645.953400:0:38:0:(import.c:2101:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff902350a11000: disconnect after 20s idle 00000100:00100000:2.0:1788201645.953424:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442788992:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201645.953519:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442789248:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201645.953552:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201645.953558:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442789120:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201645.966843:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442789376:192.168.201.130@tcp:9:kworker.0 00000100:00100000:2.0:1788201645.967170:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442789248:192.168.201.130@tcp:400:kworker.0 00000100:00080000:1.0:1788201645.993553:0:2358:0:(import.c:2011:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff90235268aa00 x1875064442789376/t0(0) o9->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 224/224 e 0 to 0 dl 1788201655 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201645.993573:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:1.0:1788201645.993581:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442789376:192.168.201.130@tcp:9:kworker.0 00010000:00010000:0.0:1788201646.787173:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201647.809490:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201648.833110:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201649.857813:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201650.882755:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201651.073447:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442789504:192.168.201.130@tcp:400:kworker.0 00000100:00000400:1.0:1788201651.073915:0:2360:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201635/real 1788201635] req@ffff9023450b9500 x1875064442788096/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201651.073953:0:2360:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9023450b9500 x1875064442788096/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201651 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201651.073963:0:2360:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:1.0:1788201651.073968:0:2360:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff902350a11000: Connection to lustre-MDT0000 (at 192.168.201.130@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788201651.099540:0:2361:0:(recover.c:205:ptlrpc_request_handle_notconn()) import MGC192.168.201.130@tcp of MGS@192.168.201.130@tcp abruptly disconnected: reconnecting 00000100:00080000:2.0:1788201651.099550:0:2361:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from FULL to DISCONN 00000100:02020000:2.0:1788201651.099558:0:2361:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail 00000100:00080000:1.0:1788201651.099933:0:2360:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00000400:1.0:1788201651.099963:0:2360:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201635/real 1788201635] req@ffff9023450ba680 x1875064442787968/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201651.099989:0:2360:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9023450ba680 x1875064442787968/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201651 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201651.102069:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201651.102083:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201651.102087:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201651.102101:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 886 00000100:00080000:3.0:1788201651.102122:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201651.102139:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201651.102189:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442789632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201651.102289:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442789760:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201651.102504:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442789632:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201651.108648:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442788096:192.168.201.130@tcp:400:kworker.0 00000100:00080000:2.0:1788201651.115396:0:2361:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.201.130@tcp for MGC192.168.201.130@tcp not replayable, auto-deactivating 00000100:00080000:2.0:1788201651.115404:0:2361:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:2.0:1788201651.115417:0:2361:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff9023450ba680 x1875064442787968/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201651 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201651.115437:0:2361:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff902352689880 x1875064442788480/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201656 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201651.115468:0:2361:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff902352688700 x1875064442788992/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201661 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201651.115479:0:2361:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff902352688a80 x1875064442789504/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201667 ref 1 fl Rpc:RNQU/200/0 rc 0/-107 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:2.0:1788201651.115489:0:2361:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1788201651.115492:0:2361:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1788201651.115494:0:2361:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:3.0:1788201651.116059:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442788992:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201651.116073:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442788480:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201651.116987:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442787968:192.168.201.130@tcp:400:kworker.0 00000100:00080000:2.0:1788201651.117062:0:2361:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201651.117070:0:2361:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 871 00000100:00080000:2.0:1788201651.117086:0:2361:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788201651.117127:0:2361:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442789888/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201651.117163:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442789504:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201651.117236:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442789888:192.168.201.130@tcp:250:ptlrpcd_01_00.0 00000100:00080000:0.0:1788201651.133076:0:2358:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.201.130@tcp: connect to target with instance 0 10000000:01000000:0.0:1788201651.133103:0:2358:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:0.0:1788201651.133114:0:2358:0:(import.c:1284:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.201.130@tcp) after server handle changed from 0x9383f985bc4d61ce to 0x9383f985bc4d64d0 00000100:00100000:3.0:1788201651.138573:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442789760:192.168.201.130@tcp:400:kworker.0 00000100:00080000:0.0:1788201651.143661:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:0.0:1788201651.143683:0:2358:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.201.130@tcp; invalidating 00000100:00100000:1.0:1788201651.145962:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442789888:192.168.201.130@tcp:250:ptlrpcd_01_00.0 00000100:00080000:0.0:1788201651.152119:0:30504:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.201.130@tcp to MGS@192.168.201.130@tcp 00000100:00100000:0.0:1788201651.152128:0:30504:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:0.0:1788201651.152135:0:30504:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:0.0:1788201651.152158:0:30504:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff90235865be00/0x4605c7c97e1c5900 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0x9383f985bc4d61dc expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201651.152173:0:30504:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff90235865be00/0x4605c7c97e1c5900 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0x9383f985bc4d61dc expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201651.152180:0:30504:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201651.152270:0:30504:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201651.152284:0:30504:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff90235865be00/0x4605c7c97e1c5900 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0x9383f985bc4d61dc expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201651.152303:0:30504:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff90235865a600/0x4605c7c97e1c590e lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x9383f985bc4d61ea expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201651.152316:0:30504:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff90235865a600/0x4605c7c97e1c590e lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x9383f985bc4d61ea expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201651.152322:0:30504:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201651.157619:0:30504:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201651.157647:0:30504:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff90235865a600/0x4605c7c97e1c590e lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x9383f985bc4d61ea expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201651.157688:0:30504:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff902345106a00/0x4605c7c97e1c58f9 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x9383f985bc4d61d5 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201651.157697:0:30504:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff902345106a00/0x4605c7c97e1c58f9 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x9383f985bc4d61d5 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201651.157704:0:30504:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201651.159500:0:30504:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201651.159519:0:30504:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff902345106a00/0x4605c7c97e1c58f9 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x9383f985bc4d61d5 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201651.159554:0:30504:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff90235865b000/0x4605c7c97e1c5907 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x9383f985bc4d61e3 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201651.159562:0:30504:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff90235865b000/0x4605c7c97e1c5907 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x9383f985bc4d61e3 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201651.159569:0:30504:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:0.0:1788201651.162543:0:30504:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201651.162568:0:30504:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff90235865b000/0x4605c7c97e1c5907 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x9383f985bc4d61e3 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1788201651.162620:0:30504:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:0.0:1788201651.162628:0:30504:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from RECOVER to FULL 10000000:01000000:0.0:1788201651.162632:0:30504:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:0.0:1788201651.166757:0:30504:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:0.0:1788201651.166778:0:30504:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.201.130@tcp: Connection restored to 192.168.201.130@tcp (at 192.168.201.130@tcp) 00010000:00010000:1.0:1788201651.905359:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201652.929180:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788201653.188152:0:5601:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788201653.188163:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-cliir-ffff902350a11000 from 21 10000000:01000000:2.0:1788201653.188171:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788201653.188241:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff902345106800/0x4605c7c97e1c5946 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201653.188257:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff902345106800/0x4605c7c97e1c5946 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201653.188271:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff902345106800/0x4605c7c97e1c5946 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201653.188308:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442790016:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201653.188573:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351938780 going to sleep for 11 seconds 00000100:00100000:3.0:1788201653.202337:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442790016:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788201653.202385:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff902345106800/0x4605c7c97e1c5946 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d64d7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201653.202404:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff902345106800/0x4605c7c97e1c5946 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d64d7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201653.202528:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442790144:192.168.201.130@tcp:256:ll_cfg_requeue.0 00000100:00100000:3.0:1788201653.202650:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b0c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201653.209205:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442790144:192.168.201.130@tcp:256:ll_cfg_requeue.0 10000000:01000000:3.0:1788201653.209277:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'lustre-cliir' failed (-2). 00010000:00010000:3.0:1788201653.209293:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff902345106800/0x4605c7c97e1c5946 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d64d7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201653.209314:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff902345106800/0x4605c7c97e1c5946 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d64d7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00000400:3.0:1788201653.209322:0:5601:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.130@tcp: IR log lustre-cliir failed, not fatal: rc = -2 10000000:01000000:3.0:1788201653.231214:0:5601:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:3.0:1788201653.231225:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-client-ffff902350a11000 from 42 10000000:01000000:3.0:1788201653.231233:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:3.0:1788201653.231305:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff902351685800/0x4605c7c97e1c594d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201653.231320:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff902351685800/0x4605c7c97e1c594d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201653.231335:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff902351685800/0x4605c7c97e1c594d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201653.231374:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442790272:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788201653.231498:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b0c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201653.234545:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442790272:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788201653.234586:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff902351685800/0x4605c7c97e1c594d lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d64de expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201653.234603:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff902351685800/0x4605c7c97e1c594d lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d64de expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201653.234734:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442790400:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788201653.234850:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b0c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201653.269588:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442790400:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788201653.269700:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442790528:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000100:00100000:3.0:1788201653.269845:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351f5b0c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201653.277329:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442790528:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000040:00080000:1.0:1788201653.278539:0:30506:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 42 00000020:01000000:3.0:1788201653.281082:0:5601:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 42-41 (rc=0) 10000000:01000000:3.0:1788201653.281127:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:3.0:1788201653.281145:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff902351685800/0x4605c7c97e1c594d lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d64de expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201653.281162:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff902351685800/0x4605c7c97e1c594d lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d64de expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201653.281286:0:5601:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:2.0:1788201653.281292:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log params-ffff902350a11000 from 7 10000000:01000000:2.0:1788201653.281296:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788201653.281362:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff902345107e00/0x4605c7c97e1c5954 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201653.281373:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff902345107e00/0x4605c7c97e1c5954 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201653.281389:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff902345107e00/0x4605c7c97e1c5954 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201653.281426:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442790656:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201653.281661:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023519383c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201653.285163:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442790656:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201653.285206:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff902345107e00/0x4605c7c97e1c5954 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d64e5 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201653.285221:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff902345107e00/0x4605c7c97e1c5954 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d64e5 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201653.285330:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442790784:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788201653.285489:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234502f300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201653.293845:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442790784:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788201653.294247:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442790912:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000100:00100000:0.0:1788201653.294389:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234502f300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201653.316105:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442790912:192.168.201.130@tcp:503:ll_cfg_requeue.0 00000040:00080000:0.0:1788201653.317245:0:30507:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000020:01000000:2.0:1788201653.317509:0:5601:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:2.0:1788201653.317533:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:2.0:1788201653.317548:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff902345107e00/0x4605c7c97e1c5954 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d64e5 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201653.317561:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff902345107e00/0x4605c7c97e1c5954 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d64e5 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201653.317582:0:5601:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:2.0:1788201653.317586:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-sptlrpc-ffff902350bf19c0 from 1 10000000:01000000:2.0:1788201653.317589:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:2.0:1788201653.317642:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff902345106e00/0x4605c7c97e1c595b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201653.317651:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff902345106e00/0x4605c7c97e1c595b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201653.317663:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff902345106e00/0x4605c7c97e1c595b lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201653.317708:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442791040:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788201653.319147:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff902351938a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201653.322268:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442791040:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:0.0:1788201653.322312:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff902345106e00/0x4605c7c97e1c595b lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d64ec expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201653.322333:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff902345106e00/0x4605c7c97e1c595b lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d64ec expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201653.322441:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442791168:192.168.201.130@tcp:501:ll_cfg_requeue.0 00000100:00100000:0.0:1788201653.322591:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234502f300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201653.325141:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442791168:192.168.201.130@tcp:501:ll_cfg_requeue.0 10000000:01000000:0.0:1788201653.325205:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:0.0:1788201653.325220:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff902345106e00/0x4605c7c97e1c595b lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d64ec expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201653.325233:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff902345106e00/0x4605c7c97e1c595b lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d64ec expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201653.953196:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201654.977799:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201656.003413:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201656.196353:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201656.196362:0:8:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788201656.196672:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442791424:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201656.196810:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201656.196816:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442791296:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201656.203855:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442791296:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201656.203918:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442791424:192.168.201.130@tcp:400:kworker.0 00000100:00000400:2.0:1788201656.257083:0:2362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201640/real 1788201640] req@ffff90235268b480 x1875064442788608/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201656 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201656.257133:0:2362:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff90235268b480 x1875064442788608/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201656 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201656.257144:0:2362:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 16, was 15): CONNECTING 00000100:00100000:2.0:1788201656.257167:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442788608:192.168.201.130@tcp:400:kworker.0 00010000:00010000:1.0:1788201657.026065:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201658.049305:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201659.075742:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201659.236640:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:2.0:1788201659.237048:0:11886:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458339584 00000100:00100000:2.0:1788201659.237062:0:11886:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458339584 00000100:00100000:2.0:1788201659.237142:0:11886:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1370 00000100:00100000:2.0:1788201659.237285:0:11886:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff90235268a680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:7ad49433-54fa-4e21-b756-859420d95603+3:24534:x1875064458339584:12345-192.168.201.130@tcp:104: 00010000:00010000:2.0:1788201659.237311:0:11886:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.201.130@tcp lock: ffff902345106800/0x4605c7c97e1c5946 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x9383f985bc4d64d7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201659.237722:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201659.237731:0:5599:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.130@tcp lock: ffff902345106800/0x4605c7c97e1c5946 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x9383f985bc4d64d7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201659.237741:0:5599:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902345106800 already unused, calling callback (ffffffffc12bbc20) 10000000:00010000:1.0:1788201659.237747:0:5599:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.130@tcp lock: ffff902345106800/0x4605c7c97e1c5946 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x9383f985bc4d64d7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201659.237756:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.130@tcp lock: ffff902345106800/0x4605c7c97e1c5946 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x9383f985bc4d64d7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788201659.237764:0:5599:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.130@tcp lock: ffff902345106800/0x4605c7c97e1c5946 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x9383f985bc4d64d7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201659.237771:0:5599:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788201659.237831:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.130@tcp lock: ffff902345106800/0x4605c7c97e1c5946 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x9383f985bc4d64d7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201659.237839:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201659.237963:0:5599:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.130@tcp lock: ffff902345106800/0x4605c7c97e1c5946 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x9383f985bc4d64d7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201659.237972:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.130@tcp lock: ffff902345106800/0x4605c7c97e1c5946 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x9383f985bc4d64d7 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201659.237997:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201659.238000:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201659.238002:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201659.238022:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442791552:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201659.238271:0:11886:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788201659.238283:0:11886:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff90235268a680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:7ad49433-54fa-4e21-b756-859420d95603+3:24534:x1875064458339584:12345-192.168.201.130@tcp:104: Request processed in 1129us (1662us total) trans 0 rc 0/0 00000100:00100000:2.0:1788201659.238297:0:11886:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1370 00000100:00100000:1.0:1788201659.248441:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442791552:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201660.097323:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788201660.353552:0:5601:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788201660.353573:0:5601:0:(mgc_request.c:1801:mgc_process_log()) Process log lustre-cliir-ffff902350a11000 from 21 10000000:01000000:2.0:1788201660.353580:0:5601:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788201660.353642:0:5601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.130@tcp lock: ffff902345106200/0x4605c7c97e1c5962 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201660.353657:0:5601:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.130@tcp lock: ffff902345106200/0x4605c7c97e1c5962 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201660.353680:0:5601:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.130@tcp lock: ffff902345106200/0x4605c7c97e1c5962 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201660.353716:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442791680:192.168.201.130@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201660.353842:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023519380c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201660.356237:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442791680:192.168.201.130@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788201660.356278:0:5601:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.130@tcp lock: ffff902345106200/0x4605c7c97e1c5962 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d666d expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201660.356299:0:5601:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.130@tcp lock: ffff902345106200/0x4605c7c97e1c5962 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x9383f985bc4d666d expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201660.356404:0:5601:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442791808:192.168.201.130@tcp:256:ll_cfg_requeue.0 00000100:00100000:2.0:1788201660.356511:0:5601:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023519380c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201660.372133:0:5601:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:7ad49433-54fa-4e21-b756-859420d95603:5601:1875064442791808:192.168.201.130@tcp:256:ll_cfg_requeue.0 00000100:00080000:0.0:1788201660.372215:0:5601:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 16, was 0): CONNECTING 00010000:00080000:0.0:1788201660.372235:0:5601:0:(ldlm_lib.c:85:import_set_conn()) imp ffff902348897000@lustre-MDT0000-mdc-ffff902350a11000: found existing conn 192.168.201.130@tcp, moved to head 10000000:01000000:0.0:1788201660.372286:0:5601:0:(mgc_request.c:1878:mgc_process_log()) MGC192.168.201.130@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:0.0:1788201660.372301:0:5601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.130@tcp lock: ffff902345106200/0x4605c7c97e1c5962 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d666d expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201660.372315:0:5601:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.130@tcp lock: ffff902345106200/0x4605c7c97e1c5962 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x9383f985bc4d666d expref: -99 pid: 5601 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201661.132508:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201661.313058:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201651/real 1788201651] req@ffff90235268bb80 x1875064442789632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201661 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201661.313106:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442789632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201661 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201661.313346:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201661.313352:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1788201661.313382:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442791936:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201661.313400:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442792064:192.168.201.130@tcp:400:kworker.0 00000100:00080000:0.0:1788201661.313512:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201661.313530:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201661.313535:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442789632:192.168.201.130@tcp:38:kworker.0 00000100:00080000:3.0:1788201661.315733:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201661.315743:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201661.315748:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201661.315759:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 0 00000100:00080000:3.0:1788201661.315770:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201661.315783:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201661.315821:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442792192/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201661.317919:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442792192:192.168.201.130@tcp:38:kworker.0 00010000:00010000:3.0:1788201661.325322:0:2362:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00100000:3.0:1788201661.325350:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442792064:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201661.325453:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788201661.325477:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201661.325479:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201661.325481:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201661.332287:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442791936:192.168.201.130@tcp:400:kworker.0 00010000:00080000:0.0:1788201661.334491:0:2358:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff902352689880 x1875064442792192/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/416 e 0 to 0 dl 1788201671 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201661.334512:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 25769803779 gen 2 00000100:00100000:0.0:1788201661.334537:0:2358:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff90234519c700 x1875064442786304/t25769803780(25769803780) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201644 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1788201661.334564:0:2358:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff902350a11000: connect to target with instance 7 00000100:00080000:0.0:1788201661.334589:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:0.0:1788201661.334598:0:2358:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788201661.334600:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 25769803779 gen 2 00000100:00100000:0.0:1788201661.334605:0:2358:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff90234519c700 x1875064442786304/t25769803780(25769803780) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201644 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1788201661.334612:0:2358:0:(recover.c:53:ptlrpc_replay_next()) import ffff902348897000 from lustre-MDT0000_UUID committed 25769803779 last 0 00000100:00080000:0.0:1788201661.334620:0:2358:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff90234519c700 x1875064442786304/t25769803780(25769803780) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201644 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201661.334634:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442792192:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788201661.334953:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442786304:192.168.201.130@tcp:101:lfs.0 00000100:00080000:1.0:1788201661.369754:0:2358:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff90234519c700 x1875064442786304/t25769803780(25769803780) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201677 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00080000:1.0:1788201661.369789:0:2358:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff9023450b8700 x1875064442787328/t25769803783(25769803783) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201644 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00020000:1.0:1788201661.369800:0:2358:0:(client.c:3435:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff90234519c700 x1875064442786304/t25769803780(25769803780) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201677 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201661.401859:0:2358:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788201661.401870:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 25769803779 gen 2 00000100:00100000:1.0:1788201661.402325:0:2358:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff90234519c700 x1875064442786304/t25769803780(25769803780) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201677 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1788201661.402346:0:2358:0:(recover.c:53:ptlrpc_replay_next()) import ffff902348897000 from lustre-MDT0000_UUID committed 25769803779 last 25769803780 00000100:00080000:1.0:1788201661.402358:0:2358:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff90234519df80 x1875064442786560/t25769803781(25769803781) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 736/824 e 0 to 0 dl 1788201644 ref 1 fl New:RQU/604/0 rc 0/301 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788201661.402373:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442786304:192.168.201.130@tcp:101:lfs.0 00000100:00100000:0.0:1788201661.402973:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442786560:192.168.201.130@tcp:101:lfs.0 00000100:00080000:0.0:1788201661.413794:0:2358:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff90234519df80 x1875064442786560/t25769803781(25769803781) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 736/608 e 0 to 0 dl 1788201677 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00080000:0.0:1788201661.413824:0:2358:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff9023450b9c00 x1875064442787072/t25769803782(25769803782) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201644 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1788201661.413833:0:2358:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788201661.413836:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 25769803779 gen 2 00000100:00100000:0.0:1788201661.413843:0:2358:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff90234519c700 x1875064442786304/t25769803780(25769803780) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201677 ref 1 fl Complete:RQU/604/0 rc 0/301 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1788201661.413853:0:2358:0:(recover.c:53:ptlrpc_replay_next()) import ffff902348897000 from lustre-MDT0000_UUID committed 25769803779 last 25769803781 00000100:00080000:0.0:1788201661.413861:0:2358:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff9023450b9c00 x1875064442787072/t25769803782(25769803782) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201644 ref 2 fl New:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201661.413918:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442786560:192.168.201.130@tcp:101:lfs.0 00000100:00100000:0.0:1788201661.413944:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442787072:192.168.201.130@tcp:35:lfs.0 00000100:00080000:0.0:1788201661.418315:0:2358:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff9023450b9c00 x1875064442787072/t25769803782(25769803782) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201677 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1788201661.418332:0:2358:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788201661.418335:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 25769803779 gen 2 00000100:00100000:0.0:1788201661.418342:0:2358:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff90234519c700 x1875064442786304/t25769803780(25769803780) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201677 ref 1 fl Complete:RQU/604/0 rc 0/301 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1788201661.418350:0:2358:0:(recover.c:53:ptlrpc_replay_next()) import ffff902348897000 from lustre-MDT0000_UUID committed 25769803779 last 25769803782 00000100:00080000:0.0:1788201661.418358:0:2358:0:(client.c:3540:ptlrpc_replay_req()) @@@ REPLAY req@ffff9023450b8700 x1875064442787328/t25769803783(25769803783) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201644 ref 2 fl New:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201661.418371:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442787072:192.168.201.130@tcp:35:lfs.0 00000100:00100000:0.0:1788201661.418394:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442787328:192.168.201.130@tcp:35:lfs.0 00000100:00080000:0.0:1788201661.421761:0:2358:0:(client.c:3425:ptlrpc_replay_interpret()) @@@ got reply req@ffff9023450b8700 x1875064442787328/t25769803783(25769803783) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201683 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1788201661.421979:0:2358:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:0.0:1788201661.421983:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 25769803779 gen 2 00000100:00100000:0.0:1788201661.421991:0:2358:0:(client.c:3156:ptlrpc_free_committed()) @@@ stopping search req@ffff90234519c700 x1875064442786304/t25769803780(25769803780) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 536/608 e 0 to 0 dl 1788201677 ref 1 fl Complete:RQU/604/0 rc 0/301 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1788201661.422006:0:2358:0:(recover.c:53:ptlrpc_replay_next()) import ffff902348897000 from lustre-MDT0000_UUID committed 25769803779 last 25769803783 00000100:00080000:0.0:1788201661.422011:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:1.0:1788201661.423612:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442787328:192.168.201.130@tcp:35:lfs.0 00010000:00080000:0.0:1788201661.423695:0:30554:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff902350a11000 to lustre-MDT0000_UUID@192.168.201.130@tcp 00010000:00010000:0.0:1788201661.423701:0:30554:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff902350a11000 (0) 00010000:00010000:0.0:1788201661.423707:0:30554:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff902350a11000 00000100:00080000:0.0:1788201661.424216:0:30554:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:1.0:1788201661.424574:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442792320:192.168.201.130@tcp:400:ldlm_lock_repla.0 00000100:00100000:1.0:1788201661.511921:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff902350a11000: committing for last_committed 25769803783 gen 2 00000100:00080000:1.0:1788201661.512036:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:1.0:1788201661.512058:0:2358:0:(client.c:3234:ptlrpc_resend_req()) @@@ going to resend req@ffff9023450b8380 x1875064442792320/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201677 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201661.512070:0:2358:0:(client.c:3243:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff9023450b8380 x1875064442792320/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201677 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201661.512077:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:0.0:1788201661.512680:0:2358:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff902350a11000: Connection restored to 192.168.201.130@tcp (at 192.168.201.130@tcp) 00000100:00100000:0.0:1788201661.512691:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442792320:192.168.201.130@tcp:400:ldlm_lock_repla.0 00010000:00010000:1.0:1788201662.146322:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1788201662.337020:0:2362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201645/real 1788201645] req@ffff902352689180 x1875064442789120/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201661 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201662.337064:0:2362:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff902352689180 x1875064442789120/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201661 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201662.337092:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442789120:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201663.169389:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201664.193058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201665.217280:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201666.241383:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201666.436076:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442792448:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201666.436173:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442792704:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201666.436203:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201666.436209:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442792576:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201666.448071:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442792576:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201666.448087:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442792448:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201666.448097:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442792704:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201667.266359:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201668.302301:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788201668.316780:0:30607:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201668.316796:0:30607:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201668.316819:0:30607:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201668.317200:0:30607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 30607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201668.317211:0:30607:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 30607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201668.317305:0:30607:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 30607 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201668.317462:0:30607:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30607:1875064442792832:192.168.201.130@tcp:101:lfs.0 00000100:00100000:2.0:1788201668.317678:0:30607:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023526f7240 going to sleep for 11 seconds 00000100:00100000:0.0:1788201668.322983:0:30607:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30607:1875064442792832:192.168.201.130@tcp:101:lfs.0 00010000:00010000:0.0:1788201668.323011:0:30607:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201668.323031:0:30607:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201668.323043:0:30607:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201668.323054:0:30607:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201668.323067:0:30607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201668.323076:0:30607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201668.323097:0:30607:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201668.323111:0:30607:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff90234598f100 x1875064442792832/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201684 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201668.323126:0:30607:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201668.323140:0:30607:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201668.323147:0:30607:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201668.323180:0:30607:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:0.0:1788201668.323187:0:30607:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5969 00000080:00010000:0.0:1788201668.323195:0:30607:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201668.323198:0:30607:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5969 from it ffffa0b181077cb8 00010000:00010000:0.0:1788201668.323202:0:30607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201668.323211:0:30607:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201668.329795:0:30607:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788201668.329807:0:30607:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201668.329833:0:30607:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201668.329851:0:30607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201668.329866:0:30607:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201668.329891:0:30607:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201668.329900:0:30607:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201668.329903:0:30607:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5969 00000080:00010000:0.0:1788201668.329911:0:30607:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5969 from it ffffa0b181077cb8 00010000:00010000:0.0:1788201668.329915:0:30607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201668.329925:0:30607:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201668.330075:0:30607:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201668.330092:0:30607:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201668.330099:0:30607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201668.330108:0:30607:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201668.330114:0:30607:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201668.330118:0:30607:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201668.330120:0:30607:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5969 00000080:00010000:0.0:1788201668.330124:0:30607:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5969 from it ffffa0b181077bb0 00010000:00010000:0.0:1788201668.330127:0:30607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201668.330134:0:30607:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201668.330143:0:30607:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 44 00000080:00200008:0.0:1788201668.330152:0:30607:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00000080:00200080:0.0:1788201668.330261:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.330277:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00800000:00000004:0.0:1788201668.330283:0:30607:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: age 901, max_age 943 00000100:00100000:0.0:1788201668.330338:0:30607:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30607:1875064442792960:192.168.201.130@tcp:41:lfs.0 00000100:00100000:2.0:1788201668.331755:0:30607:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342e36000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201668.336848:0:30607:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30607:1875064442792960:192.168.201.130@tcp:41:lfs.0 00800000:00000004:0.0:1788201668.336908:0:30607:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: update ffff902350be5190 cache blocks 17239/17271 objects 530860/531195 00000080:00200080:0.0:1788201668.336987:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337005:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337013:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337016:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337020:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337024:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337027:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337032:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337036:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337039:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337043:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337047:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337051:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337055:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337061:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337065:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337070:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337073:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337090:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337096:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337101:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337105:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337110:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337113:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337118:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337121:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337125:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337129:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337134:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337138:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337142:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337147:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337152:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337157:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337161:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337165:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337170:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337175:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337179:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337184:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337188:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337192:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337197:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337201:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337205:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337209:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337214:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337219:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337223:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337228:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337233:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337236:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337240:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337244:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337248:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337252:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337257:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337261:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337266:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337270:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337275:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337279:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337283:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337287:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337291:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337294:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337299:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337303:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337307:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337311:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337328:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337333:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337339:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337343:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337349:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337352:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337357:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337361:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337366:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337369:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337374:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337378:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337382:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337385:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337389:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337393:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337397:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337400:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337405:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337408:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337413:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337416:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337420:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337432:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337437:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337441:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337446:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337449:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337455:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337459:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337463:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337467:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337470:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337474:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337478:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337482:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337485:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.337489:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.337493:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.341722:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.341757:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.341764:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.341769:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.341773:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.341777:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.341781:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.341785:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.341789:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.341794:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.341798:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.341802:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.341806:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.341810:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.341814:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.341819:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.341822:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.341827:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00800000:00000080:0.0:1788201668.341831:0:30607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.341835:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.341841:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00020000:00000004:0.0:1788201668.341849:0:30607:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: age 901, max_age 943 00000100:00100000:0.0:1788201668.341918:0:30607:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30607:1875064442793088:192.168.201.130@tcp:13:lfs.0 00000100:00100000:0.0:1788201668.342018:0:30607:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342e36000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201668.350682:0:30607:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30607:1875064442793088:192.168.201.130@tcp:13:lfs.0 00020000:00000004:0.0:1788201668.350712:0:30607:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: update ffff902350be1e10 cache blocks 3667/3683 objects 117728/118665 00000080:00200080:0.0:1788201668.350782:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.350799:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00020000:00000004:0.0:1788201668.350806:0:30607:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: age 901, max_age 943 00000100:00080000:0.0:1788201668.350819:0:30607:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 887 00000100:00080000:0.0:1788201668.350828:0:30607:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201668.350837:0:30607:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 3407872, lost: 0. 00000100:00100000:0.0:1788201668.350861:0:30607:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442793216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201668.351460:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442793216:192.168.201.130@tcp:8:lfs.0 00000100:00080000:2.0:1788201668.352409:0:30607:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9023450ba680 x1875064442793344/t0(0) o13->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201668.352446:0:30607:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023526f7240 going to sleep for 0 seconds 00000100:00100000:1.0:1788201668.360462:0:2358:0:(client.c:3112:ptlrpc_free_committed()) lustre-OST0001-osc-ffff902350a11000: committing for last_committed 4294968873 gen 5 00000100:00080000:1.0:1788201668.360483:0:2358:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff902350a11000: connect to target with instance 1 00000100:00080000:1.0:1788201668.360493:0:2358:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff902350a11000: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788201668.360506:0:2358:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:1.0:1788201668.360510:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:1.0:1788201668.360520:0:2358:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788201668.360526:0:2358:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1788201668.360556:0:2358:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000f51a8051): req@ffff9023450ba680 x1875064442793344/t0(0) o13->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201668.360581:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442793216:192.168.201.130@tcp:8:lfs.0 00000100:00100000:2.0:1788201668.360782:0:30607:0:(client.c:2702:ptlrpc_set_wait()) set ffff9023526f7240 going to sleep for 11 seconds 00000100:00100000:0.0:1788201668.369983:0:30607:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:30607:1875064442793344:192.168.201.130@tcp:13:lfs.0 00020000:00000004:0.0:1788201668.370020:0:30607:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: update ffff902351d65190 cache blocks 3679/3683 objects 117792/118185 00000080:00200080:0.0:1788201668.370100:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370119:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370125:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370129:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370133:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370136:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370139:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370143:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370146:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370149:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370153:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370155:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370159:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370162:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370165:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370168:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370171:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370174:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370177:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370180:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370183:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370186:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370189:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370192:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370196:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370199:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370202:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370205:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370208:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370212:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370215:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370218:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370221:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370224:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370227:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370230:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370233:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370236:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370239:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370242:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370245:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370248:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370251:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370254:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370258:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370261:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370264:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370267:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370271:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370274:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370277:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370280:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370284:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370288:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370291:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370295:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370298:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370301:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370305:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370307:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370311:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370313:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370317:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370319:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370323:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370325:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370329:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370332:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370335:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370338:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370341:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370344:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370347:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370350:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370353:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370356:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370359:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370362:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370366:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370369:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370373:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370376:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370379:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370382:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370385:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370388:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370391:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370394:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370397:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370400:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370403:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370406:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370409:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370412:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370416:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370418:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370431:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370436:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370440:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370443:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370446:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370449:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370453:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370457:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370460:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370463:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370467:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370470:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370473:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370476:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370479:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370482:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370486:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370488:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370492:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370495:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370910:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370916:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370920:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370923:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370926:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370929:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370933:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370936:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200080:0.0:1788201668.370939:0:30607:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff5d7a2150 00020000:00000080:0.0:1788201668.370942:0:30607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352607000 uarg=0000000000000000 00000080:00200008:0.0:1788201668.370955:0:30607:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00010000:00010000:0.0:1788201669.313662:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201670.337823:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201671.362300:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201671.555391:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442793472:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201671.555538:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442793728:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201671.555579:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1788201671.555586:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442793600:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201671.555611:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442793856:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201671.566615:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442793600:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201671.566637:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442793856:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201671.566650:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442793472:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201671.566662:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442793728:192.168.201.130@tcp:400:kworker.0 00010000:00010000:1.0:1788201672.387901:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201673.412216:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201673.607590:0:30886:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 00010000:00010000:2.0:1788201674.433029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1788201674.991824:0:31032:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:2.0:1788201675.459180:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201676.482110:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201676.673079:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442793984:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201676.673154:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442794240:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201676.673188:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1788201676.673192:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442794112:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201676.673205:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442794368:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201676.685822:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442794112:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201676.685842:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442794368:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201676.685852:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442793984:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201676.685862:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442794240:192.168.201.130@tcp:400:kworker.0 00000080:00200008:2.0:1788201676.817720:0:31070:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201676.817736:0:31070:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201676.817759:0:31070:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201676.817777:0:31070:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201676.817793:0:31070:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201676.817802:0:31070:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201676.817809:0:31070:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201676.817813:0:31070:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5969 00000080:00010000:2.0:1788201676.817821:0:31070:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5969 from it ffffa0b180dd3cb8 00010000:00010000:2.0:1788201676.817826:0:31070:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201676.817835:0:31070:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:2.0:1788201676.820652:0:31070:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788201676.820664:0:31070:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201676.820685:0:31070:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201676.820703:0:31070:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201676.820719:0:31070:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788201676.820728:0:31070:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201676.820736:0:31070:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201676.820739:0:31070:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5969 00000080:00010000:2.0:1788201676.820746:0:31070:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5969 from it ffffa0b180dd3cb8 00010000:00010000:2.0:1788201676.820750:0:31070:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788201676.820759:0:31070:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788201676.821265:0:31070:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201676.821697:0:31070:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201676.821709:0:31070:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201676.821720:0:31070:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788201676.821728:0:31070:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201676.821733:0:31070:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201676.821737:0:31070:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5969 00000080:00010000:2.0:1788201676.821742:0:31070:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5969 from it ffffa0b180dd3bb0 00010000:00010000:2.0:1788201676.821746:0:31070:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201676.821753:0:31070:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788201676.821765:0:31070:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 44 00000080:00200008:2.0:1788201676.821774:0:31070:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00000080:00200080:2.0:1788201676.822162:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.822180:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00800000:00000004:2.0:1788201676.822187:0:31070:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: age 944, max_age 952 00000100:00100000:2.0:1788201676.822231:0:31070:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31070:1875064442794496:192.168.201.130@tcp:41:lfs.0 00000100:00100000:2.0:1788201676.822325:0:31070:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234133a6c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201676.826512:0:31070:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31070:1875064442794496:192.168.201.130@tcp:41:lfs.0 00800000:00000004:2.0:1788201676.826544:0:31070:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: update ffff902350be5190 cache blocks 17239/17271 objects 530860/531195 00000080:00200080:2.0:1788201676.826633:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826649:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826657:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826660:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826664:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826668:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826671:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826675:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826679:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826682:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826686:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826690:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826694:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826698:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826701:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826706:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826710:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826714:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826718:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826721:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826725:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826727:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826731:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826734:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826737:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826741:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826745:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826748:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826752:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826755:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826759:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826763:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826767:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826770:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826774:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826777:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826782:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826785:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826789:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826792:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826795:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826798:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826802:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826805:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826809:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826812:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826816:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826820:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826824:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826828:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826831:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826834:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826838:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826842:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826846:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826850:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826854:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826857:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826861:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826865:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826869:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826874:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826910:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826914:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826918:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826921:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826924:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826928:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826932:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826935:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826939:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826942:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826945:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826948:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826951:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826954:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826957:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826961:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826965:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826968:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826971:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826974:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826977:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826981:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826985:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826988:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.826993:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.826996:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827000:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827004:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827008:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827011:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827015:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827018:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827022:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827026:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827029:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827137:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827145:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827150:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827154:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827157:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827161:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827163:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827167:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827170:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827174:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827177:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827180:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827183:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827187:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827191:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827195:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827198:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827202:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827205:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827209:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827211:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827215:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827218:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827221:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827224:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827227:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827231:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827234:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827237:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827241:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00800000:00000080:2.0:1788201676.827244:0:31070:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.827248:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.827253:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00020000:00000004:2.0:1788201676.827261:0:31070:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: age 944, max_age 952 00000100:00100000:2.0:1788201676.827305:0:31070:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31070:1875064442794624:192.168.201.130@tcp:13:lfs.0 00000100:00100000:2.0:1788201676.827371:0:31070:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234133a6c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201676.831466:0:31070:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31070:1875064442794624:192.168.201.130@tcp:13:lfs.0 00020000:00000004:2.0:1788201676.831497:0:31070:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: update ffff902350be1e10 cache blocks 3667/3683 objects 117728/118665 00000080:00200080:2.0:1788201676.831575:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.831595:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00020000:00000004:2.0:1788201676.831603:0:31070:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: age 945, max_age 952 00000100:00100000:2.0:1788201676.831638:0:31070:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31070:1875064442794752:192.168.201.130@tcp:13:lfs.0 00000100:00100000:2.0:1788201676.831702:0:31070:0:(client.c:2702:ptlrpc_set_wait()) set ffff90234133a6c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201676.835700:0:31070:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31070:1875064442794752:192.168.201.130@tcp:13:lfs.0 00020000:00000004:2.0:1788201676.835728:0:31070:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: update ffff902351d65190 cache blocks 3679/3683 objects 117792/118185 00000080:00200080:2.0:1788201676.835797:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835813:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835820:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835824:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835828:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835831:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835836:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835839:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835843:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835847:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835851:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835854:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835859:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835862:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835866:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835869:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835908:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835914:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835920:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835925:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835929:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835933:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835937:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835940:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835944:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835948:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835951:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835955:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835959:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835963:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835967:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835971:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835975:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835979:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835983:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835986:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.835991:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.835995:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836000:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836004:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836007:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836011:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836015:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836018:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836023:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836027:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836030:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836154:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836172:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836177:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836182:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836186:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836190:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836194:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836198:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836202:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836207:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836211:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836215:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836218:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836222:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836225:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836230:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836233:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836237:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836240:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836244:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836247:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836252:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836256:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836260:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836264:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836267:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836271:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836285:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836290:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836294:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836297:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836300:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836303:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836306:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836309:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836313:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836316:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836319:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836321:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836324:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836327:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836330:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836334:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836337:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836341:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836345:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836348:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836353:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836357:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836361:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836365:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836369:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836372:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836377:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836660:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836670:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836674:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836679:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836682:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836687:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836690:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836694:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836698:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836701:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836706:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836710:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836714:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836718:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836722:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836726:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836729:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836734:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836738:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836742:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836745:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836749:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836752:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201676.836755:0:31070:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7fff71b55620 00020000:00000080:2.0:1788201676.836758:0:31070:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200008:2.0:1788201676.836771:0:31070:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00000080:00200000:1.0:1788201676.871797:0:31086:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201676.871826:0:31086:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201676.871845:0:31086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201676.871863:0:31086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1788201676.871871:0:31086:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201676.871902:0:31086:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201676.871907:0:31086:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5969 00000080:00010000:1.0:1788201676.871913:0:31086:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5969 from it ffffa0b1814f3bf8 00010000:00010000:1.0:1788201676.871918:0:31086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788201676.871927:0:31086:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788201676.871941:0:31086:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201676.871954:0:31086:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f130a.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), file ffff902343701500, open_flags 18800, mode 0 opened 0 00000080:00200000:1.0:1788201676.871960:0:31086:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f130a.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=open 00000002:00010000:1.0:1788201676.871973:0:31086:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f130a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0304001 00010000:00010000:1.0:1788201676.872022:0:31086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c5970 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 31086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201676.872030:0:31086:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c5970 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 31086 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201676.872042:0:31086:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201676.872054:0:31086:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234519df80 x1875064442795008/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 752/131944 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201676.872065:0:31086:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c5970 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 31086 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201676.872087:0:31086:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31086:1875064442795008:192.168.201.130@tcp:101:lfs.0 00000100:00100000:1.0:1788201676.872250:0:31086:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201676.877995:0:31086:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31086:1875064442795008:192.168.201.130@tcp:101:lfs.0 00010000:00010000:1.0:1788201676.878021:0:31086:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c5970 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 31086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201676.878036:0:31086:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c5970 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 31086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201676.878043:0:31086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c5970 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 31086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201676.878050:0:31086:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c5970 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 31086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201676.878060:0:31086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c5970 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 31086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201676.878067:0:31086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265ca00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201676.878072:0:31086:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c5970 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 31086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201676.878079:0:31086:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c5970 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 31086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201676.878086:0:31086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c5970 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 31086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201676.878093:0:31086:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788201676.878099:0:31086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c5970 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 31086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201676.878106:0:31086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c5970 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 31086 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201676.878128:0:31086:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=-20 req@ffff90234519df80 x1875064442795008/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 752/608 e 0 to 0 dl 1788201692 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201676.878142:0:31086:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc -20 00000080:00200000:1.0:1788201676.878327:0:31086:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201676.878343:0:31086:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201676.878357:0:31086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788201676.878368:0:31086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788201676.878374:0:31086:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201676.878499:0:31086:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201676.878503:0:31086:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5969 00000080:00010000:1.0:1788201676.878509:0:31086:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5969 from it ffffa0b1814f3bb0 00010000:00010000:1.0:1788201676.878513:0:31086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:1.0:1788201676.878522:0:31086:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788201676.878543:0:31086:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 44 00000080:00200008:1.0:1788201676.878551:0:31086:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 304000 00000080:00200080:1.0:1788201676.878578:0:31086:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c0086915 arg=19eb3d0 00000100:00100000:1.0:1788201676.878632:0:31086:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31086:1875064442795136:192.168.201.130@tcp:34:lfs.0 00000100:00100000:1.0:1788201676.878869:0:31086:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342f57300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201676.883166:0:31086:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31086:1875064442795136:192.168.201.130@tcp:34:lfs.0 00000080:00200008:1.0:1788201676.883526:0:31086:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 304000 00010000:00010000:0.0:1788201677.505635:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201678.534109:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201679.554319:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201680.578586:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201681.605723:0:10725:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 117/117 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1788201681.605979:0:5599:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 117 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201681.605991:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e5953bf8/0x4605c7c97e1b952a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8942 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.606089:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xfb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.606160:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xfb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.606242:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000089fb489f/0x4605c7c97e1b950e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a88ee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.606268:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xf9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.606305:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xf9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.606334:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000af829de2/0x4605c7c97e1b94f2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a889a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.606357:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xf7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.606378:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xf7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.606401:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000acaf5454/0x4605c7c97e1b94ba lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a87f2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.606423:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xf3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.606443:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xf3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.606465:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e4006322/0x4605c7c97e1b949e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a879e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.606489:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xf1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.606507:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xf1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.606530:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ba0d0969/0x4605c7c97e1b9482 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x76:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a874a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.606552:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xef:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.606572:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xef:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.606595:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008ce5c00f/0x4605c7c97e1b9466 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a86f6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.606619:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xed:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.606639:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xed:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.606661:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000095064702/0x4605c7c97e1b944a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x74:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a86a2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.606683:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xeb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.606703:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xeb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.606725:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000020ab3c9/0x4605c7c97e1b942e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a864e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.606748:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xe9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.606773:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xe9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.606796:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000018edfedd/0x4605c7c97e1b9412 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a85fa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.606820:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xe7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.606839:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xe7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.606861:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007cce4b9e/0x4605c7c97e1b93f6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a85a6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.606924:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xe5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.606945:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xe5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.606970:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000078071ee8/0x4605c7c97e1b93da lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8552 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.606991:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xe3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.607012:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xe3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.607035:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c02b2460/0x4605c7c97e1b93a2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a84aa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.607057:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xdf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.607075:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xdf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.607098:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008d5f19e7/0x4605c7c97e1b9386 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8456 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.607122:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xdd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.607141:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xdd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.607164:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000db9517f1/0x4605c7c97e1b936a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8402 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.607186:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xdb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.607218:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xdb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.607242:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000328ac31a/0x4605c7c97e1b934e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a83ae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.607265:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xd9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.607290:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xd9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.607313:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000de73776f/0x4605c7c97e1b9332 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a835a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.607335:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xd7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.607355:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xd7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.607379:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e2255a66/0x4605c7c97e1b8e9a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x40:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7592 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.607402:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x83:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.607423:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x83:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.607448:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006f20d85a/0x4605c7c97e1b9316 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8306 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.607473:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xd5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.607492:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xd5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.607515:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f09eff5f/0x4605c7c97e1b92fa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a82b2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.607537:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xd3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.607558:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xd3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.607583:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000033a48d1b/0x4605c7c97e1b92de lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a825e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.607606:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xd1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.607627:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xd1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.607650:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000734f8693/0x4605c7c97e1b92c2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a820a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.607671:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xcf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.607689:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xcf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.607710:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006060837a/0x4605c7c97e1b92a6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a81b6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.607731:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xcd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.607756:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xcd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.607776:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000094133d83/0x4605c7c97e1b928a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8162 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.607797:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xcb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.607813:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xcb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.607835:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000137fa104/0x4605c7c97e1b926e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a810e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.607855:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xc9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.607871:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.607922:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000250a8b35/0x4605c7c97e1b9252 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a80ba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.607945:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xc7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.607967:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.607989:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007158ef55/0x4605c7c97e1b9236 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x61:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8066 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.608013:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xc5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.608032:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.608057:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000382482ca/0x4605c7c97e1b921a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x60:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8012 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.608081:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xc3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.608100:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.608124:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000040df7b40/0x4605c7c97e1b91fe lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x5f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7fbe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.608149:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xc1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.608169:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xc1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.608206:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006cfe2d4/0x4605c7c97e1b91c6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x5d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7f16 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.608232:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xbd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.608257:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xbd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.608280:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000048ddea66/0x4605c7c97e1b91aa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x5c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7ec2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.608303:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xbb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.608322:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xbb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.608345:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ddb3f5cb/0x4605c7c97e1b918e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x5b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7e6e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.608366:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xb9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.608386:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xb9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.608410:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000159413ab/0x4605c7c97e1b9172 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x5a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7e1a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.608432:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xb7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.608452:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xb7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.608476:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000456c880f/0x4605c7c97e1b9156 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x59:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7dc6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.608501:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xb5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.608522:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xb5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.608544:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c693379e/0x4605c7c97e1b913a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x58:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7d72 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.608567:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xb3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.608587:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xb3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.608611:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000043d4aaf1/0x4605c7c97e1b8d82 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x36:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a724a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.608635:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x6f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.608656:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x6f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.608682:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005c28bf4d/0x4605c7c97e1b911e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x57:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7d1e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.608706:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xb1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.608731:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xb1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.608755:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000741dee66/0x4605c7c97e1b9102 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x56:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7cca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.608780:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xaf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.608803:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xaf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.608828:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a853ef38/0x4605c7c97e1b90ca lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x54:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7c22 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.608853:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xab:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.608872:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xab:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.608925:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ac9dc1e9/0x4605c7c97e1b90ae lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x53:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7bce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.608950:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xa9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.608971:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xa9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.608995:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000126b16b4/0x4605c7c97e1b9092 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x52:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7b7a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.609018:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xa7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.609040:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xa7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.609065:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001a62e565/0x4605c7c97e1b9076 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x51:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7b26 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.609089:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xa5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.609111:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xa5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.609135:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000311dfac9/0x4605c7c97e1b905a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x50:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7ad2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.609159:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xa3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.609178:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xa3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201681.610855:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201681.619156:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000699e55fd/0x4605c7c97e1b903e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x4f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7a7e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.619258:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xa1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.619495:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xa1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.619530:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000019bd01f3/0x4605c7c97e1b9022 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x4e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7a2a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.619557:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.619579:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.619602:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000073fe83f3/0x4605c7c97e1b9006 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x4d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a79d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.619627:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.619651:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.619675:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006b6007a2/0x4605c7c97e1b8fea lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x4c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7982 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.619696:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.620415:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.620456:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011a9611a/0x4605c7c97e1b8fb2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x4a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a78da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.620491:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x97:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.620519:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x97:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.620545:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d858dc83/0x4605c7c97e1b8f96 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x49:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7886 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.620569:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x95:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.620590:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x95:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.620614:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000020c02ba9/0x4605c7c97e1b8f7a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x48:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7832 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.620639:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x93:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.620660:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x93:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.620684:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000064b37b50/0x4605c7c97e1b8f5e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x47:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a77de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.620708:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x91:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.621090:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x91:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.621118:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fb63cfd8/0x4605c7c97e1b8f42 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x46:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a778a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.621145:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x8f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.621162:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x8f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.621185:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000055843d70/0x4605c7c97e1b8c6a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x2c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6f02 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.621221:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x5b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.621247:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x5b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.621271:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e89fdfba/0x4605c7c97e1b8f26 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x45:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7736 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.621295:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x8d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.622218:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x8d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.622255:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009c70d35a/0x4605c7c97e1b8f0a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x44:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a76e2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.622295:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x8b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.622472:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x8b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.622498:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a3063877/0x4605c7c97e1b8eee lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a768e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.622524:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x89:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.622545:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x89:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.622569:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000032968989/0x4605c7c97e1b8ed2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a763a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.622593:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x87:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.622614:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x87:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.622638:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001d5e62fb/0x4605c7c97e1b8eb6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x41:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a75e6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.622662:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x85:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.622683:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x85:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.622706:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000db6a51d0/0x4605c7c97e1b8e7e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x3f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a753e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.623027:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x81:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.623048:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x81:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.623123:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000caea6075/0x4605c7c97e1b8e62 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x3e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a74ea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.623148:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x7f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.623171:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x7f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.623202:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7634d5a/0x4605c7c97e1b8e46 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x3d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7496 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.623224:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x7d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.623241:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x7d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.623261:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a79c8a10/0x4605c7c97e1b8e2a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x3c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7442 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.623281:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x7b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.623365:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x7b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.623392:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ce0cab73/0x4605c7c97e1b8e0e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x3b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a73ee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.623416:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x79:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.623436:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x79:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.623458:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007af2f8fc/0x4605c7c97e1b8df2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x3a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a739a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.623480:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x77:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.623500:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x77:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.623524:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000082e5c4a9/0x4605c7c97e1b8dd6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x39:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7346 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.623548:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x75:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.623568:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x75:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.623590:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f99761bd/0x4605c7c97e1b8dba lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x38:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a72f2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.623613:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x73:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.623632:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x73:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.623656:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000489ab3ff/0x4605c7c97e1b8d9e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x37:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a729e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.623678:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x71:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.623703:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x71:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.625003:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000052c52ea0/0x4605c7c97e1b8d66 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x35:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a71f6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.625054:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x6d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.626003:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x6d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.626052:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000018a2c939/0x4605c7c97e1b8d4a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x34:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a71a2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.626273:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x6b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.626318:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x6b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.626344:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c10b23be/0x4605c7c97e1b8d2e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x33:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a714e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.626368:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x69:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.626388:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x69:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.626413:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000036884dbf/0x4605c7c97e1b8d12 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x32:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a70fa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.626435:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x67:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.626454:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x67:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.626477:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000270bee66/0x4605c7c97e1b8cf6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x31:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a70a6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.626500:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x65:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.626521:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x65:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.626545:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e553ab7c/0x4605c7c97e1b8cda lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x30:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7052 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.626569:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x63:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.626590:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x63:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.626615:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c2ec018b/0x4605c7c97e1b8cbe lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x2f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6ffe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.626638:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x61:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.626665:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x61:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.626688:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000094e00f79/0x4605c7c97e1b8ca2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x2e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6faa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.626710:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x5f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.628400:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x5f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.628458:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009d3bc942/0x4605c7c97e1b8c86 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x2d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6f56 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.628512:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x5d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.628533:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x5d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.628557:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000030c9b28d/0x4605c7c97e1b8c4e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x2b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6eae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.628582:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x59:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.628601:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x59:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.628627:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002dbcdf47/0x4605c7c97e1b8c32 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x2a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6e5a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.628651:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x57:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.628672:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x57:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.628695:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000edb0459f/0x4605c7c97e1b8c16 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x29:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6e06 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.629386:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x55:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.629422:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x55:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.629457:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000092cd62dd/0x4605c7c97e1b8bfa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6db2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.629485:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x53:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.629506:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x53:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.629530:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000887aca8e/0x4605c7c97e1b8bde lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x27:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6d5e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.629554:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x51:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.629581:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x51:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.629605:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007a76ab63/0x4605c7c97e1b8bc2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x26:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6d0a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.629629:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x4f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.629648:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x4f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.629671:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bb3035f8/0x4605c7c97e1b8ba6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x25:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6cb6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.629694:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x4d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.629727:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x4d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.629754:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e5c6c131/0x4605c7c97e1b8b8a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x24:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6c62 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.629778:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x4b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.629797:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x4b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.629821:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fe9e9aea/0x4605c7c97e1b8b6e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x23:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6c0e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.629846:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x49:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.629865:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x49:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.629908:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007c52a2c3/0x4605c7c97e1b8b52 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6bba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.629930:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x47:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.629949:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x47:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.629973:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005ef5d9ab/0x4605c7c97e1b8b36 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x21:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6b66 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.629995:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x45:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.630013:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x45:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.630035:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000027dab088/0x4605c7c97e1b8b1a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x20:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6b12 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.630058:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x43:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.630093:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x43:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.630117:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000060c5ab58/0x4605c7c97e1b8afe lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6abe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.630141:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x41:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.630160:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x41:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.630182:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a26fa36f/0x4605c7c97e1b8ae2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6a6a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.630217:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.630237:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.630261:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cff9c5b9/0x4605c7c97e1b8ac6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6a16 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.630283:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.630301:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.630324:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004e5b7cad/0x4605c7c97e1b8aaa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a69c2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.630347:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.630365:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.630389:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d116f232/0x4605c7c97e1b8a8e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a696e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.630413:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x39:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.630433:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x39:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.630455:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d174e57e/0x4605c7c97e1b8a72 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a691a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.630478:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x37:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.630496:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x37:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.630523:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b3b26b6f/0x4605c7c97e1b8a56 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x19:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a68c6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.630549:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x35:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.630573:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x35:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.630595:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d0389090/0x4605c7c97e1b8a3a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6872 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.630618:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x33:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.630636:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x33:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.630659:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ebff234c/0x4605c7c97e1b8a1e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x17:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a681e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.630680:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x31:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.630697:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x31:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.630718:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e7263853/0x4605c7c97e1b8a02 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x16:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a67ca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.630739:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.630758:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.630781:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a9fa1205/0x4605c7c97e1b89e6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x15:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6776 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.630804:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.630823:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.630846:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ec3ac1b7/0x4605c7c97e1b89ca lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x14:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6722 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.630870:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.630922:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.630947:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000552cc7eb/0x4605c7c97e1b89ae lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x13:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a66ce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.630972:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x29:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.630991:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x29:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.631014:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d6234611/0x4605c7c97e1b8992 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x12:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a667a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.631038:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x27:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.631060:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x27:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.631085:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fe40ccc9/0x4605c7c97e1b8976 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x11:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6626 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.631109:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x25:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.631130:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x25:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.631154:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008009f862/0x4605c7c97e1b895a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x10:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a65d2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.631177:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x23:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.633274:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x23:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.633476:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e3e4f58e/0x4605c7c97e1b893e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a657e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.633534:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x21:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.633584:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x21:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.633607:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008c1ca5ac/0x4605c7c97e1b8922 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a652a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.633631:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.633653:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.633677:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000deb66e35/0x4605c7c97e1b8906 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a64d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.633707:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.633742:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.633768:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000082f06df9/0x4605c7c97e1b88ea lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6482 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.633791:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.633814:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.633839:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e2b288cf/0x4605c7c97e1b88ce lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a642e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.633863:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x19:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.633918:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x19:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.633942:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000027de8108/0x4605c7c97e1b88b2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xa:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a63da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.633965:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x17:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.633983:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x17:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.634005:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004edb9dc6/0x4605c7c97e1b8896 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6386 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.634028:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x15:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.634048:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x15:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.634073:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000030dc1d44/0x4605c7c97e1b887a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6332 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.634110:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x13:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.634128:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x13:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.634150:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007efe311d/0x4605c7c97e1b885e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a62de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.634172:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x11:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.634189:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x11:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.634222:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006ff4850/0x4605c7c97e1b8842 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a628a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.634248:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.634268:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.634292:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000592b8bba/0x4605c7c97e1b8826 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a6236 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.634325:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.634344:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.634366:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c71c49c0/0x4605c7c97e1b880a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a61e2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.634386:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.634404:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.634425:0:5599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2516531/0x4605c7c97e1b87ee lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a618e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201681.634446:0:5599:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201681.634463:0:5599:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1788201681.634522:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e5953bf8/0x4605c7c97e1b952a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8942 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634531:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000089fb489f/0x4605c7c97e1b950e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a88ee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634538:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000af829de2/0x4605c7c97e1b94f2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a889a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634546:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000acaf5454/0x4605c7c97e1b94ba lrc: 1/0,0 mode: --/PW res: [0x240000400:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a87f2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634553:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e4006322/0x4605c7c97e1b949e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a879e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634559:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ba0d0969/0x4605c7c97e1b9482 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x76:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a874a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634567:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008ce5c00f/0x4605c7c97e1b9466 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a86f6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634574:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000095064702/0x4605c7c97e1b944a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x74:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a86a2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634581:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000020ab3c9/0x4605c7c97e1b942e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a864e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634589:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000018edfedd/0x4605c7c97e1b9412 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a85fa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634597:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007cce4b9e/0x4605c7c97e1b93f6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a85a6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634605:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000078071ee8/0x4605c7c97e1b93da lrc: 1/0,0 mode: --/PW res: [0x240000400:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8552 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634616:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c02b2460/0x4605c7c97e1b93a2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a84aa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634625:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008d5f19e7/0x4605c7c97e1b9386 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8456 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634633:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000db9517f1/0x4605c7c97e1b936a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8402 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634640:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000328ac31a/0x4605c7c97e1b934e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a83ae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634647:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000de73776f/0x4605c7c97e1b9332 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a835a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634655:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e2255a66/0x4605c7c97e1b8e9a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x40:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7592 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634663:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006f20d85a/0x4605c7c97e1b9316 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8306 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634670:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f09eff5f/0x4605c7c97e1b92fa lrc: 1/0,0 mode: --/PW res: [0x240000400:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a82b2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634677:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000033a48d1b/0x4605c7c97e1b92de lrc: 1/0,0 mode: --/PW res: [0x240000400:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a825e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634688:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000734f8693/0x4605c7c97e1b92c2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a820a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634694:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006060837a/0x4605c7c97e1b92a6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a81b6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634702:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000094133d83/0x4605c7c97e1b928a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8162 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634709:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000137fa104/0x4605c7c97e1b926e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a810e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634716:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000250a8b35/0x4605c7c97e1b9252 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a80ba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634723:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007158ef55/0x4605c7c97e1b9236 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x61:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8066 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634730:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000382482ca/0x4605c7c97e1b921a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x60:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8012 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634737:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000040df7b40/0x4605c7c97e1b91fe lrc: 1/0,0 mode: --/PW res: [0x240000400:0x5f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7fbe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634744:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006cfe2d4/0x4605c7c97e1b91c6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x5d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7f16 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634755:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000048ddea66/0x4605c7c97e1b91aa lrc: 1/0,0 mode: --/PW res: [0x240000400:0x5c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7ec2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634762:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ddb3f5cb/0x4605c7c97e1b918e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x5b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7e6e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634769:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000159413ab/0x4605c7c97e1b9172 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x5a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7e1a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634777:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000456c880f/0x4605c7c97e1b9156 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x59:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7dc6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634785:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c693379e/0x4605c7c97e1b913a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x58:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7d72 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634794:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000043d4aaf1/0x4605c7c97e1b8d82 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x36:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a724a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634802:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005c28bf4d/0x4605c7c97e1b911e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x57:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7d1e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634811:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000741dee66/0x4605c7c97e1b9102 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x56:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7cca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634818:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a853ef38/0x4605c7c97e1b90ca lrc: 1/0,0 mode: --/PW res: [0x240000400:0x54:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7c22 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634833:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ac9dc1e9/0x4605c7c97e1b90ae lrc: 1/0,0 mode: --/PW res: [0x240000400:0x53:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7bce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634841:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000126b16b4/0x4605c7c97e1b9092 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x52:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7b7a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634849:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001a62e565/0x4605c7c97e1b9076 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x51:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7b26 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634857:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000311dfac9/0x4605c7c97e1b905a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x50:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7ad2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634865:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000699e55fd/0x4605c7c97e1b903e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x4f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7a7e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634873:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000019bd01f3/0x4605c7c97e1b9022 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x4e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7a2a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634915:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000073fe83f3/0x4605c7c97e1b9006 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x4d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a79d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634922:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006b6007a2/0x4605c7c97e1b8fea lrc: 1/0,0 mode: --/PW res: [0x240000400:0x4c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7982 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634929:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011a9611a/0x4605c7c97e1b8fb2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x4a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a78da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634940:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d858dc83/0x4605c7c97e1b8f96 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x49:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7886 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634947:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000020c02ba9/0x4605c7c97e1b8f7a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x48:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7832 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634953:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000064b37b50/0x4605c7c97e1b8f5e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x47:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a77de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634960:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fb63cfd8/0x4605c7c97e1b8f42 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x46:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a778a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634967:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000055843d70/0x4605c7c97e1b8c6a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x2c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6f02 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634974:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e89fdfba/0x4605c7c97e1b8f26 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x45:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7736 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634982:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009c70d35a/0x4605c7c97e1b8f0a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x44:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a76e2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634989:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a3063877/0x4605c7c97e1b8eee lrc: 1/0,0 mode: --/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a768e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.634997:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000032968989/0x4605c7c97e1b8ed2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a763a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635009:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001d5e62fb/0x4605c7c97e1b8eb6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x41:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a75e6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635016:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000db6a51d0/0x4605c7c97e1b8e7e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x3f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a753e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635024:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000caea6075/0x4605c7c97e1b8e62 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x3e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a74ea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635032:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7634d5a/0x4605c7c97e1b8e46 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x3d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7496 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635039:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a79c8a10/0x4605c7c97e1b8e2a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x3c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7442 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635046:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ce0cab73/0x4605c7c97e1b8e0e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x3b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a73ee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635054:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007af2f8fc/0x4605c7c97e1b8df2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x3a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a739a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635061:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000082e5c4a9/0x4605c7c97e1b8dd6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x39:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7346 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635068:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f99761bd/0x4605c7c97e1b8dba lrc: 1/0,0 mode: --/PW res: [0x240000400:0x38:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a72f2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635079:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000489ab3ff/0x4605c7c97e1b8d9e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x37:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a729e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635086:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000052c52ea0/0x4605c7c97e1b8d66 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x35:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a71f6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635093:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000018a2c939/0x4605c7c97e1b8d4a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x34:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a71a2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635101:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c10b23be/0x4605c7c97e1b8d2e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x33:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a714e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635109:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000036884dbf/0x4605c7c97e1b8d12 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x32:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a70fa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635116:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000270bee66/0x4605c7c97e1b8cf6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x31:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a70a6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635124:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e553ab7c/0x4605c7c97e1b8cda lrc: 1/0,0 mode: --/PW res: [0x240000400:0x30:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7052 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635131:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c2ec018b/0x4605c7c97e1b8cbe lrc: 1/0,0 mode: --/PW res: [0x240000400:0x2f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6ffe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635139:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000094e00f79/0x4605c7c97e1b8ca2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x2e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6faa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635150:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009d3bc942/0x4605c7c97e1b8c86 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x2d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6f56 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635157:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000030c9b28d/0x4605c7c97e1b8c4e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x2b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6eae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635165:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002dbcdf47/0x4605c7c97e1b8c32 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x2a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6e5a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635172:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000edb0459f/0x4605c7c97e1b8c16 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x29:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6e06 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635179:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000092cd62dd/0x4605c7c97e1b8bfa lrc: 1/0,0 mode: --/PW res: [0x240000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6db2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635186:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000887aca8e/0x4605c7c97e1b8bde lrc: 1/0,0 mode: --/PW res: [0x240000400:0x27:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6d5e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635204:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007a76ab63/0x4605c7c97e1b8bc2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x26:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6d0a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635213:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bb3035f8/0x4605c7c97e1b8ba6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x25:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6cb6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635220:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e5c6c131/0x4605c7c97e1b8b8a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x24:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6c62 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635231:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fe9e9aea/0x4605c7c97e1b8b6e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x23:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6c0e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635238:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007c52a2c3/0x4605c7c97e1b8b52 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6bba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635245:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005ef5d9ab/0x4605c7c97e1b8b36 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x21:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6b66 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635251:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000027dab088/0x4605c7c97e1b8b1a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x20:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6b12 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635258:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000060c5ab58/0x4605c7c97e1b8afe lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6abe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635265:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a26fa36f/0x4605c7c97e1b8ae2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6a6a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635272:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cff9c5b9/0x4605c7c97e1b8ac6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6a16 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635279:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004e5b7cad/0x4605c7c97e1b8aaa lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a69c2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635287:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d116f232/0x4605c7c97e1b8a8e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a696e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635299:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d174e57e/0x4605c7c97e1b8a72 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a691a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635306:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b3b26b6f/0x4605c7c97e1b8a56 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x19:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a68c6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635313:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d0389090/0x4605c7c97e1b8a3a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6872 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635321:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ebff234c/0x4605c7c97e1b8a1e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x17:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a681e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635330:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e7263853/0x4605c7c97e1b8a02 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x16:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a67ca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635337:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a9fa1205/0x4605c7c97e1b89e6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x15:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6776 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635345:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ec3ac1b7/0x4605c7c97e1b89ca lrc: 1/0,0 mode: --/PW res: [0x240000400:0x14:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6722 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635353:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000552cc7eb/0x4605c7c97e1b89ae lrc: 1/0,0 mode: --/PW res: [0x240000400:0x13:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a66ce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635361:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d6234611/0x4605c7c97e1b8992 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x12:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a667a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635372:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fe40ccc9/0x4605c7c97e1b8976 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x11:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6626 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635380:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008009f862/0x4605c7c97e1b895a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x10:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a65d2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635388:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e3e4f58e/0x4605c7c97e1b893e lrc: 1/0,0 mode: --/PW res: [0x240000400:0xf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a657e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635396:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008c1ca5ac/0x4605c7c97e1b8922 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a652a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635402:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000deb66e35/0x4605c7c97e1b8906 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a64d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635410:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000082f06df9/0x4605c7c97e1b88ea lrc: 1/0,0 mode: --/PW res: [0x240000400:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6482 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635416:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e2b288cf/0x4605c7c97e1b88ce lrc: 1/0,0 mode: --/PW res: [0x240000400:0xb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a642e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635424:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000027de8108/0x4605c7c97e1b88b2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xa:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a63da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635431:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004edb9dc6/0x4605c7c97e1b8896 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6386 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635443:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000030dc1d44/0x4605c7c97e1b887a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6332 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635449:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007efe311d/0x4605c7c97e1b885e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a62de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635456:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006ff4850/0x4605c7c97e1b8842 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a628a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635462:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000592b8bba/0x4605c7c97e1b8826 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6236 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635468:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c71c49c0/0x4605c7c97e1b880a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a61e2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635475:0:5599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2516531/0x4605c7c97e1b87ee lrc: 1/0,0 mode: --/PW res: [0x240000400:0x3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a618e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635481:0:5599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 117 locks packed 00010000:00010000:2.0:1788201681.635534:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e5953bf8/0x4605c7c97e1b952a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x7c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8942 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635561:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000089fb489f/0x4605c7c97e1b950e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x7b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a88ee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635572:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000af829de2/0x4605c7c97e1b94f2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x7a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a889a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635583:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000acaf5454/0x4605c7c97e1b94ba lrc: 0/0,0 mode: --/PW res: [0x240000400:0x78:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a87f2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635593:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e4006322/0x4605c7c97e1b949e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x77:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a879e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635604:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ba0d0969/0x4605c7c97e1b9482 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x76:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a874a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635617:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008ce5c00f/0x4605c7c97e1b9466 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x75:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a86f6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635628:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000095064702/0x4605c7c97e1b944a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x74:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a86a2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635640:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000020ab3c9/0x4605c7c97e1b942e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x73:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a864e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635651:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000018edfedd/0x4605c7c97e1b9412 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x72:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a85fa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635661:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007cce4b9e/0x4605c7c97e1b93f6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x71:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a85a6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635677:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000078071ee8/0x4605c7c97e1b93da lrc: 0/0,0 mode: --/PW res: [0x240000400:0x70:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8552 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635687:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c02b2460/0x4605c7c97e1b93a2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x6e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a84aa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635698:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008d5f19e7/0x4605c7c97e1b9386 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x6d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8456 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635708:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000db9517f1/0x4605c7c97e1b936a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8402 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635719:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000328ac31a/0x4605c7c97e1b934e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x6b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a83ae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635729:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000de73776f/0x4605c7c97e1b9332 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a835a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635739:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e2255a66/0x4605c7c97e1b8e9a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x40:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7592 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635752:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006f20d85a/0x4605c7c97e1b9316 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x69:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8306 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635769:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f09eff5f/0x4605c7c97e1b92fa lrc: 0/0,0 mode: --/PW res: [0x240000400:0x68:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a82b2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635779:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000033a48d1b/0x4605c7c97e1b92de lrc: 0/0,0 mode: --/PW res: [0x240000400:0x67:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a825e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635790:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000734f8693/0x4605c7c97e1b92c2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x66:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a820a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635803:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006060837a/0x4605c7c97e1b92a6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x65:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a81b6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635814:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000094133d83/0x4605c7c97e1b928a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x64:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8162 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635825:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000137fa104/0x4605c7c97e1b926e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x63:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a810e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635835:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000250a8b35/0x4605c7c97e1b9252 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x62:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a80ba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635847:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007158ef55/0x4605c7c97e1b9236 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x61:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8066 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635864:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000382482ca/0x4605c7c97e1b921a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x60:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8012 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635902:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000040df7b40/0x4605c7c97e1b91fe lrc: 0/0,0 mode: --/PW res: [0x240000400:0x5f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7fbe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635914:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006cfe2d4/0x4605c7c97e1b91c6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x5d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7f16 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635926:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000048ddea66/0x4605c7c97e1b91aa lrc: 0/0,0 mode: --/PW res: [0x240000400:0x5c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7ec2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635938:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ddb3f5cb/0x4605c7c97e1b918e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x5b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7e6e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635950:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000159413ab/0x4605c7c97e1b9172 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x5a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7e1a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635961:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000456c880f/0x4605c7c97e1b9156 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x59:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7dc6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635974:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c693379e/0x4605c7c97e1b913a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x58:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7d72 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.635991:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000043d4aaf1/0x4605c7c97e1b8d82 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x36:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a724a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.636003:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005c28bf4d/0x4605c7c97e1b911e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x57:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7d1e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.636015:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000741dee66/0x4605c7c97e1b9102 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x56:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7cca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.636028:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a853ef38/0x4605c7c97e1b90ca lrc: 0/0,0 mode: --/PW res: [0x240000400:0x54:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7c22 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.636039:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ac9dc1e9/0x4605c7c97e1b90ae lrc: 0/0,0 mode: --/PW res: [0x240000400:0x53:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7bce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.636050:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000126b16b4/0x4605c7c97e1b9092 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x52:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7b7a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.636061:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001a62e565/0x4605c7c97e1b9076 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x51:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7b26 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.636070:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000311dfac9/0x4605c7c97e1b905a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x50:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7ad2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.636085:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000699e55fd/0x4605c7c97e1b903e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x4f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7a7e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.636095:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000019bd01f3/0x4605c7c97e1b9022 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x4e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7a2a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.636105:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000073fe83f3/0x4605c7c97e1b9006 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x4d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a79d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.636115:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006b6007a2/0x4605c7c97e1b8fea lrc: 0/0,0 mode: --/PW res: [0x240000400:0x4c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7982 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.636126:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011a9611a/0x4605c7c97e1b8fb2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x4a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a78da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.636137:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d858dc83/0x4605c7c97e1b8f96 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x49:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7886 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.636148:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000020c02ba9/0x4605c7c97e1b8f7a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x48:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7832 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.636157:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000064b37b50/0x4605c7c97e1b8f5e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x47:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a77de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.636173:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fb63cfd8/0x4605c7c97e1b8f42 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x46:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a778a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.636186:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000055843d70/0x4605c7c97e1b8c6a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x2c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6f02 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640528:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e89fdfba/0x4605c7c97e1b8f26 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x45:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7736 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640551:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009c70d35a/0x4605c7c97e1b8f0a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x44:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a76e2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640565:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a3063877/0x4605c7c97e1b8eee lrc: 0/0,0 mode: --/PW res: [0x240000400:0x43:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a768e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640579:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000032968989/0x4605c7c97e1b8ed2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x42:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a763a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640592:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001d5e62fb/0x4605c7c97e1b8eb6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x41:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a75e6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640606:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000db6a51d0/0x4605c7c97e1b8e7e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x3f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a753e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640625:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000caea6075/0x4605c7c97e1b8e62 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x3e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a74ea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640636:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7634d5a/0x4605c7c97e1b8e46 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x3d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7496 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640646:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a79c8a10/0x4605c7c97e1b8e2a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x3c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7442 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640657:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ce0cab73/0x4605c7c97e1b8e0e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x3b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a73ee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640667:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007af2f8fc/0x4605c7c97e1b8df2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x3a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a739a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640680:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000082e5c4a9/0x4605c7c97e1b8dd6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x39:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7346 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640694:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f99761bd/0x4605c7c97e1b8dba lrc: 0/0,0 mode: --/PW res: [0x240000400:0x38:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a72f2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640705:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000489ab3ff/0x4605c7c97e1b8d9e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x37:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a729e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640738:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000052c52ea0/0x4605c7c97e1b8d66 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x35:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a71f6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640750:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000018a2c939/0x4605c7c97e1b8d4a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x34:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a71a2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640762:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c10b23be/0x4605c7c97e1b8d2e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x33:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a714e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640774:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000036884dbf/0x4605c7c97e1b8d12 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x32:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a70fa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640786:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000270bee66/0x4605c7c97e1b8cf6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x31:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a70a6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640797:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e553ab7c/0x4605c7c97e1b8cda lrc: 0/0,0 mode: --/PW res: [0x240000400:0x30:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7052 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640810:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c2ec018b/0x4605c7c97e1b8cbe lrc: 0/0,0 mode: --/PW res: [0x240000400:0x2f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6ffe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640823:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000094e00f79/0x4605c7c97e1b8ca2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x2e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6faa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640841:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009d3bc942/0x4605c7c97e1b8c86 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x2d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6f56 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640852:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000030c9b28d/0x4605c7c97e1b8c4e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x2b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6eae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.640864:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002dbcdf47/0x4605c7c97e1b8c32 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x2a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6e5a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1788201681.642032:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442795264:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201681.642272:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000edb0459f/0x4605c7c97e1b8c16 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x29:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6e06 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651228:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000092cd62dd/0x4605c7c97e1b8bfa lrc: 0/0,0 mode: --/PW res: [0x240000400:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6db2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651273:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000887aca8e/0x4605c7c97e1b8bde lrc: 0/0,0 mode: --/PW res: [0x240000400:0x27:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6d5e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651287:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007a76ab63/0x4605c7c97e1b8bc2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x26:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6d0a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651312:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bb3035f8/0x4605c7c97e1b8ba6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x25:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6cb6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651332:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e5c6c131/0x4605c7c97e1b8b8a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x24:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6c62 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651345:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fe9e9aea/0x4605c7c97e1b8b6e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x23:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6c0e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651358:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007c52a2c3/0x4605c7c97e1b8b52 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6bba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651369:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005ef5d9ab/0x4605c7c97e1b8b36 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x21:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6b66 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651381:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000027dab088/0x4605c7c97e1b8b1a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x20:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6b12 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651394:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000060c5ab58/0x4605c7c97e1b8afe lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6abe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651407:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a26fa36f/0x4605c7c97e1b8ae2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6a6a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651418:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cff9c5b9/0x4605c7c97e1b8ac6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6a16 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651436:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004e5b7cad/0x4605c7c97e1b8aaa lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a69c2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651449:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d116f232/0x4605c7c97e1b8a8e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a696e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651462:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d174e57e/0x4605c7c97e1b8a72 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a691a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651475:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b3b26b6f/0x4605c7c97e1b8a56 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x19:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a68c6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651488:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d0389090/0x4605c7c97e1b8a3a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x18:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6872 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651502:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ebff234c/0x4605c7c97e1b8a1e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x17:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a681e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651515:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e7263853/0x4605c7c97e1b8a02 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x16:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a67ca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651527:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a9fa1205/0x4605c7c97e1b89e6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x15:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6776 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651545:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ec3ac1b7/0x4605c7c97e1b89ca lrc: 0/0,0 mode: --/PW res: [0x240000400:0x14:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6722 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651556:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000552cc7eb/0x4605c7c97e1b89ae lrc: 0/0,0 mode: --/PW res: [0x240000400:0x13:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a66ce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651567:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d6234611/0x4605c7c97e1b8992 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x12:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a667a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651579:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fe40ccc9/0x4605c7c97e1b8976 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x11:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6626 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651592:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008009f862/0x4605c7c97e1b895a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x10:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a65d2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651605:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e3e4f58e/0x4605c7c97e1b893e lrc: 0/0,0 mode: --/PW res: [0x240000400:0xf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a657e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651616:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008c1ca5ac/0x4605c7c97e1b8922 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a652a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651628:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000deb66e35/0x4605c7c97e1b8906 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a64d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651645:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000082f06df9/0x4605c7c97e1b88ea lrc: 0/0,0 mode: --/PW res: [0x240000400:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6482 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651658:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e2b288cf/0x4605c7c97e1b88ce lrc: 0/0,0 mode: --/PW res: [0x240000400:0xb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a642e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651671:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000027de8108/0x4605c7c97e1b88b2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xa:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a63da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651683:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004edb9dc6/0x4605c7c97e1b8896 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6386 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651696:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000030dc1d44/0x4605c7c97e1b887a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6332 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651708:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007efe311d/0x4605c7c97e1b885e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a62de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651721:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006ff4850/0x4605c7c97e1b8842 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a628a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651734:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000592b8bba/0x4605c7c97e1b8826 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a6236 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651753:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c71c49c0/0x4605c7c97e1b880a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a61e2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201681.651764:0:5599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2516531/0x4605c7c97e1b87ee lrc: 0/0,0 mode: --/PW res: [0x240000400:0x3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a618e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201681.651940:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201681.651945:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201681.651948:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201681.662447:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442795264:192.168.201.130@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788201681.794109:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442795392:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201681.794203:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442795648:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201681.794242:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201681.794248:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442795520:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201681.807794:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442795520:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201681.807812:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7ad49433-54fa-4e21-b756-859420d95603:2360:1875064442795392:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201681.807822:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442795648:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201682.626166:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788201683.070840:0:31652:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 130b: DoM file create (inherited) replay ========================================================== 14:41:21 (1788201681) 00000080:00200000:3.0:1788201683.457593:0:31667:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201683.457623:0:31667:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201683.457644:0:31667:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788201683.457661:0:31667:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:3.0:1788201683.457674:0:31667:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201683.457682:0:31667:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201683.457686:0:31667:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5969 00000080:00010000:3.0:1788201683.457693:0:31667:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5969 from it ffffa0b181c83b58 00010000:00010000:3.0:1788201683.457698:0:31667:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788201683.457708:0:31667:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1788201683.457724:0:31667:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201683.457740:0:31667:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=13 00000080:00200000:3.0:1788201683.457744:0:31667:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=getattr 00000002:00010000:3.0:1788201683.457758:0:31667:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d130b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201683.457805:0:31667:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c5977 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 31667 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201683.457814:0:31667:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c5977 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 31667 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201683.457824:0:31667:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c5977 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 31667 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201683.457850:0:31667:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31667:1875064442795776:192.168.201.130@tcp:101:lfs.0 00000100:00100000:3.0:1788201683.459620:0:31667:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342e36a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201683.465957:0:31667:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31667:1875064442795776:192.168.201.130@tcp:101:lfs.0 00010000:00010000:0.0:1788201683.465981:0:31667:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c5977 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 31667 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.465994:0:31667:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c5977 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 31667 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.466002:0:31667:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c5977 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 31667 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.466013:0:31667:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c5977 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 31667 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.466023:0:31667:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c5977 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 31667 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.466030:0:31667:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff902351684400 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201683.466035:0:31667:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c5977 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 31667 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.466041:0:31667:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c5977 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 31667 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.466049:0:31667:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c5977 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 31667 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.466055:0:31667:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201683.466062:0:31667:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c5977 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 31667 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.466068:0:31667:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351684400/0x4605c7c97e1c5977 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 31667 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201683.466086:0:31667:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff902352689180 x1875064442795776/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201699 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201683.466101:0:31667:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1788201683.466126:0:31667:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:0.0:1788201683.466150:0:31667:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00010000:0.0:1788201683.466166:0:31667:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5969 from it ffffa0b181c83930 00010000:00010000:0.0:1788201683.466171:0:31667:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:0.0:1788201683.466178:0:31667:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:0.0:1788201683.466255:0:31667:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201683.466270:0:31667:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201683.466277:0:31667:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:0.0:1788201683.466286:0:31667:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:0.0:1788201683.466292:0:31667:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201683.466298:0:31667:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201683.466301:0:31667:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5969 00000080:00010000:0.0:1788201683.466306:0:31667:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5969 from it ffffa0b181c83be8 00010000:00010000:0.0:1788201683.466309:0:31667:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:0.0:1788201683.466316:0:31667:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:0.0:1788201683.466324:0:31667:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:0.0:1788201683.466331:0:31667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d130b.replay-single, flags=16392 00000080:00200008:0.0:1788201683.466358:0:31667:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788201683.466363:0:31667:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201683.466370:0:31667:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201683.466377:0:31667:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788201683.466385:0:31667:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:0.0:1788201683.466392:0:31667:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201683.466397:0:31667:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201683.466399:0:31667:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5969 00000080:00010000:0.0:1788201683.466403:0:31667:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5969 from it ffffa0b181c83cb8 00010000:00010000:0.0:1788201683.466406:0:31667:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788201683.466414:0:31667:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200008:0.0:1788201683.466977:0:31667:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788201683.466981:0:31667:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201683.466989:0:31667:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201683.466998:0:31667:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1788201683.467006:0:31667:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:0.0:1788201683.467012:0:31667:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201683.467017:0:31667:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201683.467019:0:31667:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5969 00000080:00010000:0.0:1788201683.467023:0:31667:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5969 from it ffffa0b181c83cb8 00010000:00010000:0.0:1788201683.467026:0:31667:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1788201683.467033:0:31667:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:0.0:1788201683.467146:0:31667:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:0.0:1788201683.467154:0:31667:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201683.467173:0:31667:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:0.0:1788201683.467181:0:31667:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:0.0:1788201683.467187:0:31667:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201683.467192:0:31667:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:0.0:1788201683.467195:0:31667:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5969 00000080:00010000:0.0:1788201683.467199:0:31667:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5969 from it ffffa0b181c83bb0 00010000:00010000:0.0:1788201683.467201:0:31667:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:0.0:1788201683.467208:0:31667:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:0.0:1788201683.467215:0:31667:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 44 00000080:00200008:0.0:1788201683.467221:0:31667:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 300000 00000080:00200080:0.0:1788201683.467236:0:31667:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c03066f0 arg=7ffdab3e5b90 00000080:00200000:0.0:1788201683.467249:0:31667:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) name=d130b.replay-single stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:0.0:1788201683.467273:0:31667:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:0.0:1788201683.467323:0:31667:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:0.0:1788201683.467331:0:31667:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201683.467334:0:31667:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902345106a00/0x4605c7c97e1c5969 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d686c expref: -99 pid: 30607 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000020:00100000:0.0:1788201683.467358:0:31667:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201683.467367:0:31667:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450bb100 x1875064442796032/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 568/664 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201683.467390:0:31667:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31667:1875064442796032:192.168.201.130@tcp:36:lfs.0 00000100:00100000:0.0:1788201683.467478:0:31667:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358723cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201683.485056:0:31667:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb100 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31667:1875064442796032:192.168.201.130@tcp:36:lfs.0 00000080:00200000:0.0:1788201683.485114:0:31667:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9023449b6c08) 00000080:00200000:0.0:1788201683.485122:0:31667:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x42c:0x0], updating i_size 512 00000080:00200008:0.0:1788201683.485174:0:31667:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 300000 00000080:00200000:1.0:1788201683.508338:0:31668:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201683.508390:0:31668:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201683.508456:0:31668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.508467:0:31668:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.508478:0:31668:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201683.508505:0:31668:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31668:1875064442796160:192.168.201.130@tcp:101:lfs.0 00000100:00100000:1.0:1788201683.508593:0:31668:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2900 going to sleep for 11 seconds 00000100:00100000:1.0:1788201683.512576:0:31668:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31668:1875064442796160:192.168.201.130@tcp:101:lfs.0 00010000:00010000:1.0:1788201683.512608:0:31668:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.512621:0:31668:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.512631:0:31668:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.512643:0:31668:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.512660:0:31668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.512669:0:31668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.512675:0:31668:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201683.512688:0:31668:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff90234519c380 x1875064442796160/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201699 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201683.513413:0:31668:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.513434:0:31668:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201683.513442:0:31668:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201683.513472:0:31668:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:1.0:1788201683.513478:0:31668:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c597e 00000080:00010000:1.0:1788201683.513484:0:31668:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201683.513488:0:31668:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c597e from it ffffa0b181ca3b58 00010000:00010000:1.0:1788201683.513491:0:31668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.513500:0:31668:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201683.513530:0:31668:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201683.513551:0:31668:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=8 00000080:00200000:1.0:1788201683.513556:0:31668:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=getattr 00000002:00010000:1.0:1788201683.513574:0:31668:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d130b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201683.513616:0:31668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.513623:0:31668:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.513632:0:31668:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201683.513652:0:31668:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31668:1875064442796288:192.168.201.130@tcp:101:lfs.0 00000100:00100000:1.0:1788201683.513911:0:31668:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2900 going to sleep for 11 seconds 00000100:00100000:1.0:1788201683.524145:0:31668:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31668:1875064442796288:192.168.201.130@tcp:101:lfs.0 00010000:00010000:1.0:1788201683.524182:0:31668:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.524205:0:31668:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.524267:0:31668:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.524282:0:31668:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.524294:0:31668:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.524308:0:31668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.524316:0:31668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.524324:0:31668:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201683.524335:0:31668:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff90234519c380 x1875064442796288/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201699 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201683.524347:0:31668:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.524355:0:31668:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1068 (0 0) 00000080:00200000:1.0:1788201683.524362:0:31668:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201683.524383:0:31668:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x42c:0x0], updating i_size 512 00000080:00200000:1.0:1788201683.524386:0:31668:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x42c:0x0](ffff9023449b6c08): rc = 0 00000080:00010000:1.0:1788201683.524391:0:31668:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42c:0x0](ffff9023449b6c08) for lock 0x4605c7c97e1c5985 00000080:00010000:1.0:1788201683.524402:0:31668:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42c:0x0](ffff9023449b6c08) 00000080:00010000:1.0:1788201683.524406:0:31668:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5985 from it ffffa0b181ca3b20 00010000:00010000:1.0:1788201683.524409:0:31668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201683.524423:0:31668:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788201683.524457:0:31668:0:(file.c:6283:ll_getattr_dentry()) START file d130b.replay-single:[0x200000bd1:0x42c:0x0](ffff9023449b6c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201683.524463:0:31668:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9023449b6c08),name=d130b.replay-single 00000002:00010000:1.0:1788201683.524481:0:31668:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x42c:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201683.524491:0:31668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201683.524501:0:31668:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201683.524508:0:31668:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201683.524512:0:31668:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42c:0x0](ffff9023449b6c08) 00000080:00010000:1.0:1788201683.524515:0:31668:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42c:0x0](ffff9023449b6c08) for lock 0x4605c7c97e1c5985 00000080:00010000:1.0:1788201683.524519:0:31668:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5985 from it ffffa0b181ca3cb8 00010000:00010000:1.0:1788201683.524522:0:31668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201683.524530:0:31668:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201683.524586:0:31668:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201683.524594:0:31668:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201683.524601:0:31668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201683.524615:0:31668:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201683.524621:0:31668:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201683.524625:0:31668:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201683.524627:0:31668:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c597e 00000080:00010000:1.0:1788201683.524631:0:31668:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c597e from it ffffa0b181ca3bf8 00010000:00010000:1.0:1788201683.524634:0:31668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201683.524641:0:31668:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201683.524649:0:31668:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201683.524656:0:31668:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d130b.replay-single, flags=267 00000080:00200000:1.0:1788201683.524662:0:31668:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9023449b6c08), inode mode 41ed mask 44 00000080:00200008:1.0:1788201683.524670:0:31668:0:(file.c:1090:ll_file_open()) START file d130b.replay-single:[0x200000bd1:0x42c:0x0](ffff9023449b6c08), flags 300000 00000002:00010000:1.0:1788201683.524684:0:31668:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x42c:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0240300001 00010000:00010000:0.0:1788201683.528347:0:31668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c598c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.528365:0:31668:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c598c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201683.528388:0:31668:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201683.528400:0:31668:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234519c380 x1875064442796544/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 736/131944 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201683.528409:0:31668:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c598c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201683.528435:0:31668:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31668:1875064442796544:192.168.201.130@tcp:101:lfs.0 00000100:00100000:0.0:1788201683.528636:0:31668:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358723cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201683.567408:0:31668:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31668:1875064442796544:192.168.201.130@tcp:101:lfs.0 00010000:00010000:0.0:1788201683.567451:0:31668:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c598c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.567470:0:31668:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c598c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.567477:0:31668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c598c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.567484:0:31668:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c598c lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.567494:0:31668:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c598c lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.567500:0:31668:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235865aa00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:0.0:1788201683.567505:0:31668:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c598c lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.567516:0:31668:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c598c lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.567523:0:31668:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c598c lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.567531:0:31668:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201683.567537:0:31668:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c598c lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201683.567544:0:31668:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235865aa00/0x4605c7c97e1c598c lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 31668 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201683.567565:0:31668:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff90234519c380 x1875064442796544/t30064771074(30064771074) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 736/608 e 0 to 0 dl 1788201699 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201683.567580:0:31668:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90234519c380 x1875064442796544/t30064771074(30064771074) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 736/608 e 0 to 0 dl 1788201699 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201683.567603:0:31668:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201683.567630:0:31668:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x42c:0x0], updating i_size 512 00000002:00100000:0.0:1788201683.567650:0:31668:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90234519c380 x1875064442796544/t30064771074(30064771074) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 736/608 e 0 to 0 dl 1788201699 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201683.567762:0:31668:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9023449b6c08), inode mode 41ed mask 2 00000080:00200000:0.0:1788201683.567774:0:31668:0:(xattr.c:413:ll_xattr_set()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9023449b6c08), xattr lov 00010000:00010000:0.0:1788201683.567797:0:31668:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201683.567847:0:31668:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201683.567854:0:31668:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201683.567857:0:31668:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265d600/0x4605c7c97e1c5985 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d68f8 expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201683.569036:0:31668:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201683.569052:0:31668:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450b9880 x1875064442796800/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 680/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201683.569080:0:31668:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31668:1875064442796800:192.168.201.130@tcp:36:lfs.0 00000100:00100000:0.0:1788201683.569332:0:31668:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358723840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201683.582196:0:31668:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9880 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31668:1875064442796800:192.168.201.130@tcp:36:lfs.0 00000080:00200008:0.0:1788201683.582263:0:31668:0:(file.c:479:ll_file_release()) START file d130b.replay-single:[0x200000bd1:0x42c:0x0](ffff9023449b6c08), flags 300000 00000002:00100000:0.0:1788201683.582332:0:31668:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff90234519c380 x1875064442796544/t30064771074(30064771074) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 736/608 e 0 to 0 dl 1788201699 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201683.582350:0:31668:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201683.582356:0:31668:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9023450b9500 x1875064442797056/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/712 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201683.582390:0:31668:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31668:1875064442797056:192.168.201.130@tcp:35:lfs.0 00000100:00100000:0.0:1788201683.585899:0:31668:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358723cc0 going to sleep for 17 seconds 00000100:00100000:0.0:1788201683.609088:0:31668:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31668:1875064442797056:192.168.201.130@tcp:35:lfs.0 00000002:00100000:0.0:1788201683.609127:0:31668:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff90234519c380 x1875064442796544/t30064771074(30064771074) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 736/608 e 0 to 0 dl 1788201699 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788201683.650018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788201684.566190:0:31677:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201684.566206:0:31677:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201684.566230:0:31677:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201684.566249:0:31677:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201684.566266:0:31677:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201684.566280:0:31677:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201684.566288:0:31677:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201684.566292:0:31677:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c597e 00000080:00010000:1.0:1788201684.566299:0:31677:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c597e from it ffffa0b181c83cb8 00010000:00010000:1.0:1788201684.566304:0:31677:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201684.566314:0:31677:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200008:1.0:1788201684.570223:0:31677:0:(file.c:6283:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff902352108908), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788201684.570234:0:31677:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201684.570256:0:31677:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201684.570274:0:31677:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201684.570290:0:31677:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788201684.570299:0:31677:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201684.570307:0:31677:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201684.570311:0:31677:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c597e 00000080:00010000:1.0:1788201684.570318:0:31677:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c597e from it ffffa0b181c83cb8 00010000:00010000:1.0:1788201684.570322:0:31677:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201684.570338:0:31677:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201684.570665:0:31677:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201684.570675:0:31677:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201684.570684:0:31677:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201684.570693:0:31677:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788201684.570700:0:31677:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201684.570705:0:31677:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201684.570708:0:31677:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c597e 00000080:00010000:1.0:1788201684.570712:0:31677:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c597e from it ffffa0b181c83bb0 00010000:00010000:1.0:1788201684.570715:0:31677:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788201684.570724:0:31677:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788201684.570733:0:31677:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 44 00000080:00200008:1.0:1788201684.570742:0:31677:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00000080:00200080:1.0:1788201684.571421:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:1.0:1788201684.571442:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff90234581f400 uarg=0000000000000000 00800000:00000004:1.0:1788201684.571450:0:31677:0:(obd_class.h:1091:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: age 953, max_age 960 00000100:00100000:1.0:1788201684.571494:0:31677:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d500 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31677:1875064442797184:192.168.201.130@tcp:41:lfs.0 00000100:00100000:1.0:1788201684.571595:0:31677:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2e40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201684.588040:0:31677:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d500 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31677:1875064442797184:192.168.201.130@tcp:41:lfs.0 00800000:00000004:1.0:1788201684.588069:0:31677:0:(obd_class.h:1117:obd_statfs()) lustre-MDT0000-mdc-ffff902350a11000: update ffff902350be5190 cache blocks 17239/17271 objects 530860/531195 00000080:00200080:2.0:1788201684.591806:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.591836:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.591848:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.591853:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.591858:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.591862:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.591867:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.591871:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.591898:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.591904:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.591908:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.591912:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.591915:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.591919:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.591923:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.591926:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.591930:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.591933:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.591938:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.591941:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.591945:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.591948:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.591952:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.591955:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.591960:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.591963:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.591967:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.591970:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.591974:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.591978:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.591982:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.591985:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.591989:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.591993:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.591997:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592000:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592004:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592019:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592024:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592029:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592033:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592037:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592041:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592045:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592049:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592053:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592057:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592060:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592064:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592067:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592071:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592075:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592079:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592082:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592085:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592088:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592091:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592094:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592098:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592102:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592105:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592109:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592113:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592116:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592120:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592124:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592128:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592131:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592135:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592138:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592153:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592157:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592161:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592164:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592168:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592171:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592175:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592178:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592182:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592185:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592189:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592192:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592196:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592200:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592204:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592208:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592212:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592215:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592219:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592222:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592226:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592230:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592234:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592238:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592242:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592479:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592489:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592495:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592500:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592504:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592508:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592511:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592515:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592518:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592522:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592525:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592528:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592531:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592534:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592537:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592541:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592544:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592548:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592551:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592555:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592559:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592563:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592566:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592571:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592574:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592578:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592581:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592585:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592589:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592593:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592597:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592601:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00800000:00000080:2.0:1788201684.592605:0:31677:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00000080:00200080:2.0:1788201684.592609:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:2.0:1788201684.592615:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902345192800 uarg=0000000000000000 00020000:00000004:2.0:1788201684.592623:0:31677:0:(obd_class.h:1091:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: age 953, max_age 960 00000100:00100000:2.0:1788201684.592670:0:31677:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31677:1875064442797312:192.168.201.130@tcp:13:lfs.0 00000100:00100000:2.0:1788201684.593342:0:31677:0:(client.c:2702:ptlrpc_set_wait()) set ffff902343438e40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201684.600767:0:31677:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31677:1875064442797312:192.168.201.130@tcp:13:lfs.0 00020000:00000004:0.0:1788201684.600807:0:31677:0:(obd_class.h:1117:obd_statfs()) lustre-OST0000-osc-ffff902350a11000: update ffff902350be1e10 cache blocks 3667/3683 objects 117728/118665 00000080:00200080:0.0:1788201684.600912:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.600928:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00020000:00000004:0.0:1788201684.600936:0:31677:0:(obd_class.h:1091:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: age 953, max_age 960 00000100:00100000:0.0:1788201684.600970:0:31677:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31677:1875064442797440:192.168.201.130@tcp:13:lfs.0 00000100:00100000:0.0:1788201684.601038:0:31677:0:(client.c:2702:ptlrpc_set_wait()) set ffff902358723cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201684.608665:0:31677:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job lfs:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31677:1875064442797440:192.168.201.130@tcp:13:lfs.0 00020000:00000004:0.0:1788201684.608696:0:31677:0:(obd_class.h:1117:obd_statfs()) lustre-OST0001-osc-ffff902350a11000: update ffff902351d65190 cache blocks 3679/3683 objects 117792/118185 00000080:00200080:0.0:1788201684.608767:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.608859:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.608870:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.608911:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.608919:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.608924:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.608929:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.608932:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.608936:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.608939:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.608944:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.608947:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.608951:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.608954:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.608958:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.608962:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.608965:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.608969:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.608973:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.608976:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.608980:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.608983:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.608986:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.608989:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.608992:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.608996:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.608999:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609003:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609007:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609106:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609113:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609116:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609120:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609124:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609128:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609131:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609135:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609138:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609152:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609156:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609160:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609163:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609167:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609171:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609174:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609178:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609181:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609184:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609188:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609191:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609195:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609198:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609202:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609205:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609208:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609211:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609215:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609218:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609222:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609225:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609229:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609232:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609235:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609468:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609476:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609481:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609485:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609488:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609492:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609495:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609499:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609503:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609506:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609509:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609513:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609516:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609520:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609523:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609527:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609530:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609533:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609537:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609540:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609544:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609548:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609551:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609555:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609559:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609562:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609566:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609570:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609573:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609577:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609581:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609585:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609588:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609592:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609596:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609600:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609604:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609608:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609612:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609615:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609618:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609623:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609626:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609630:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609634:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609637:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609640:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609644:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609647:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609651:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609655:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609659:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609662:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609666:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609669:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609673:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609677:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609681:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609685:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609689:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609692:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200080:0.0:1788201684.609696:0:31677:0:(dir.c:2003:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908) cmd=c00866a4 arg=7ffcbf2ec960 00020000:00000080:0.0:1788201684.609699:0:31677:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff902350a11000: cmd=c00866a4 len=592 karg=ffff902352606800 uarg=0000000000000000 00000080:00200008:0.0:1788201684.609714:0:31677:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff902352108908), flags 100000 00010000:00010000:2.0:1788201684.677937:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788201684.712524:0:31684:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201684.712553:0:31684:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201684.712571:0:31684:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201684.712588:0:31684:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788201684.712597:0:31684:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201684.712605:0:31684:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201684.712610:0:31684:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c597e 00000080:00010000:2.0:1788201684.712618:0:31684:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c597e from it ffffa0b181cc3bd0 00010000:00010000:2.0:1788201684.712622:0:31684:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788201684.712630:0:31684:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788201684.712646:0:31684:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201684.712659:0:31684:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg130-client.virtnet, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=1536 00000080:00200000:2.0:1788201684.712663:0:31684:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201684.712669:0:31684:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201684.712675:0:31684:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201684.712682:0:31684:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788201684.712687:0:31684:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201684.712691:0:31684:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201684.712693:0:31684:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c597e 00000080:00010000:2.0:1788201684.712697:0:31684:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c597e from it ffffa0b181cc3cd0 00010000:00010000:2.0:1788201684.712700:0:31684:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788201684.712706:0:31684:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1788201684.712718:0:31684:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 3 00000080:00200000:2.0:1788201684.712723:0:31684:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201684.712729:0:31684:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201684.712734:0:31684:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788201684.712741:0:31684:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1788201684.712747:0:31684:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201684.712750:0:31684:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201684.712753:0:31684:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c597e 00000080:00010000:2.0:1788201684.712756:0:31684:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c597e from it ffffa0b181cc3db0 00010000:00010000:2.0:1788201684.712759:0:31684:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788201684.712765:0:31684:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788201684.712773:0:31684:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 3 00000080:00200000:2.0:1788201684.712778:0:31684:0:(namei.c:2129:ll_create_nd()) VFS Op:name=fsa-oleg130-client.virtnet, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=33188, excl=1 00000080:00200000:2.0:1788201684.712782:0:31684:0:(namei.c:2084:ll_mknod()) VFS Op:name=fsa-oleg130-client.virtnet, dir=[0x200000007:0x1:0x0](ffff902352108908) mode 100644 dev 0 00010000:00010000:2.0:1788201684.712807:0:31684:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788201684.712852:0:31684:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788201684.712860:0:31684:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201684.712863:0:31684:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c400/0x4605c7c97e1c597e lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d68dc expref: -99 pid: 31668 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000020:00100000:2.0:1788201684.712914:0:31684:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201684.712928:0:31684:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234598ea00 x1875064442797696/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 528/664 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201684.712953:0:31684:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ea00 pname:cluuid:pid:xid:nid:opc:job mcreate:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31684:1875064442797696:192.168.201.130@tcp:36:mcreate.0 00000100:00100000:2.0:1788201684.713061:0:31684:0:(client.c:2702:ptlrpc_set_wait()) set ffff902343438840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201684.728442:0:31684:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ea00 pname:cluuid:pid:xid:nid:opc:job mcreate:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31684:1875064442797696:192.168.201.130@tcp:36:mcreate.0 00000080:00200000:0.0:1788201684.728497:0:31684:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x42d:0x0](ffff9023449b2a08) 00000080:00200000:0.0:1788201684.728579:0:31684:0:(namei.c:2137:ll_create_nd()) VFS Op:name=fsa-oleg130-client.virtnet, unhashed 1 00000080:00200000:3.0:1788201684.753940:0:31687:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:3.0:1788201684.753970:0:31687:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201684.754041:0:31687:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 31687 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201684.754052:0:31687:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 31687 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201684.754062:0:31687:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 31687 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201684.754087:0:31687:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31687:1875064442797824:192.168.201.130@tcp:101:rm.0 00000100:00100000:3.0:1788201684.754191:0:31687:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342e36600 going to sleep for 11 seconds 00000100:00100000:3.0:1788201684.759760:0:31687:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31687:1875064442797824:192.168.201.130@tcp:101:rm.0 00010000:00010000:3.0:1788201684.759803:0:31687:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201684.759816:0:31687:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201684.759828:0:31687:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201684.759841:0:31687:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201684.759853:0:31687:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201684.759862:0:31687:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201684.760112:0:31687:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201684.760128:0:31687:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352688700 x1875064442797824/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201700 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201684.760155:0:31687:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201684.760167:0:31687:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201684.760175:0:31687:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201684.760205:0:31687:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:3.0:1788201684.760212:0:31687:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5993 00000080:00010000:3.0:1788201684.760218:0:31687:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:3.0:1788201684.760221:0:31687:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5993 from it ffffa0b181bf3b58 00010000:00010000:3.0:1788201684.760225:0:31687:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201684.760233:0:31687:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201684.760272:0:31687:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:3.0:1788201684.760290:0:31687:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg130-client.virtnet, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=8 00000080:00200000:3.0:1788201684.760295:0:31687:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg130-client.virtnet, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=getattr 00000002:00010000:3.0:1788201684.760315:0:31687:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: fsa-oleg130-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201684.760354:0:31687:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 31687 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201684.760376:0:31687:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 31687 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201684.760386:0:31687:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 31687 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201684.760406:0:31687:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31687:1875064442797952:192.168.201.130@tcp:101:rm.0 00000100:00100000:3.0:1788201684.760484:0:31687:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342e36600 going to sleep for 11 seconds 00000100:00100000:3.0:1788201684.772432:0:31687:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31687:1875064442797952:192.168.201.130@tcp:101:rm.0 00010000:00010000:3.0:1788201684.772465:0:31687:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201684.775655:0:31687:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201684.775676:0:31687:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201684.775688:0:31687:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201684.775700:0:31687:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201684.775719:0:31687:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201684.775727:0:31687:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201684.775739:0:31687:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201684.775756:0:31687:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff902352688700 x1875064442797952/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 600/584 e 0 to 0 dl 1788201700 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201684.775770:0:31687:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201684.775794:0:31687:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1069 (0 0) 00000080:00200000:2.0:1788201684.775802:0:31687:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201684.775823:0:31687:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x42d:0x0], updating i_size 0 00000080:00200000:2.0:1788201684.775827:0:31687:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x42d:0x0](ffff9023449b2a08): rc = 0 00000080:00200000:2.0:1788201684.775870:0:31687:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x42d:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1788201684.775909:0:31687:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42d:0x0](ffff9023449b2a08) for lock 0x4605c7c97e1c599a 00000080:00010000:2.0:1788201684.775919:0:31687:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42d:0x0](ffff9023449b2a08) 00000080:00010000:2.0:1788201684.775923:0:31687:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c599a from it ffffa0b181bf3b20 00010000:00010000:2.0:1788201684.775926:0:31687:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201684.775935:0:31687:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788201684.775970:0:31687:0:(file.c:6283:ll_getattr_dentry()) START file fsa-oleg130-client.virtnet:[0x200000bd1:0x42d:0x0](ffff9023449b2a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788201684.775977:0:31687:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000bd1:0x42d:0x0](ffff9023449b2a08),name=fsa-oleg130-client.virtnet 00000002:00010000:2.0:1788201684.775994:0:31687:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000bd1:0x42d:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201684.776003:0:31687:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201684.776013:0:31687:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 2/1,0 mode: PR/PR res: [0x200000bd1:0x42d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201684.776020:0:31687:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201684.776035:0:31687:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42d:0x0](ffff9023449b2a08) 00000080:00010000:2.0:1788201684.776038:0:31687:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42d:0x0](ffff9023449b2a08) for lock 0x4605c7c97e1c599a 00000080:00010000:2.0:1788201684.776041:0:31687:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c599a from it ffffa0b181bf3cb8 00010000:00010000:2.0:1788201684.776044:0:31687:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201684.776052:0:31687:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201684.776110:0:31687:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201684.776118:0:31687:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201684.776125:0:31687:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201684.776135:0:31687:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201684.776152:0:31687:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201684.776156:0:31687:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201684.776159:0:31687:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5993 00000080:00010000:2.0:1788201684.776162:0:31687:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5993 from it ffffa0b181bf3c10 00010000:00010000:2.0:1788201684.776166:0:31687:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201684.776172:0:31687:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201684.776181:0:31687:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201684.776190:0:31687:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg130-client.virtnet, flags=0 00000080:00200000:2.0:1788201684.776195:0:31687:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:2.0:1788201684.776201:0:31687:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201684.776206:0:31687:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201684.776214:0:31687:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201684.776220:0:31687:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201684.776223:0:31687:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:2.0:1788201684.776226:0:31687:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c5993 00000080:00010000:2.0:1788201684.776229:0:31687:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c5993 from it ffffa0b181bf3d68 00010000:00010000:2.0:1788201684.776232:0:31687:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201684.776239:0:31687:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201684.776248:0:31687:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 3 00000080:00200000:2.0:1788201684.776253:0:31687:0:(namei.c:2418:ll_unlink()) VFS Op:name=fsa-oleg130-client.virtnet, dir=[0x200000007:0x1:0x0](ffff902352108908) 00010000:00010000:2.0:1788201684.776288:0:31687:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201684.776313:0:31687:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201684.776331:0:31687:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x42d:0x0]: losing layout lock 00010000:00010000:2.0:1788201684.776368:0:31687:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x42d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201684.776375:0:31687:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788201684.776380:0:31687:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201684.776383:0:31687:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685200/0x4605c7c97e1c599a lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x42d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d6945 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201684.776400:0:31687:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff902351685000/0x4605c7c97e1c5993 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d6929 expref: -99 pid: 31687 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:2.0:1788201684.776420:0:31687:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201684.776430:0:31687:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234598c000 x1875064442798208/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 512/712 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201684.776454:0:31687:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c000 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31687:1875064442798208:192.168.201.130@tcp:36:rm.0 00000100:00100000:2.0:1788201684.776556:0:31687:0:(client.c:2702:ptlrpc_set_wait()) set ffff902343438540 going to sleep for 11 seconds 00000100:00100000:0.0:1788201684.781218:0:31687:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c000 pname:cluuid:pid:xid:nid:opc:job rm:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31687:1875064442798208:192.168.201.130@tcp:36:rm.0 00000080:00200000:0.0:1788201684.781322:0:31687:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x42d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1788201684.781331:0:31687:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x42d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201684.781357:0:31687:0:(llite_lib.c:2075:ll_clear_inode()) VFS Op:inode=[0x200000bd1:0x42d:0x0](ffff9023449b2a08) 00010000:00010000:3.0:1788201685.697549:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201686.721249:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201686.915034:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442798464:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201686.915151:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442798720:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201686.915217:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442798336:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201686.915258:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442798592:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201686.919444:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442798592:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201686.919767:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442798464:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201686.919779:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442798720:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201686.925869:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442798336:192.168.201.130@tcp:400:kworker.0 00000001:02000400:2.0:1788201687.723523:0:31718:0:(debug.c:654:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00000080:00200000:1.0:1788201687.731339:0:31719:0:(file.c:6178:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908),name=/ 00000002:00010000:1.0:1788201687.731466:0:31719:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201687.731514:0:31719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c59a1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 31719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.731522:0:31719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c59a1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 31719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.731529:0:31719:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c59a1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 31719 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201687.731548:0:31719:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d500 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31719:1875064442798848:192.168.201.130@tcp:101:touch.0 00000100:00100000:1.0:1788201687.731744:0:31719:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2d80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201687.734640:0:31719:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d500 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31719:1875064442798848:192.168.201.130@tcp:101:touch.0 00010000:00010000:1.0:1788201687.734672:0:31719:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c59a1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d696f expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.734684:0:31719:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c59a1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d696f expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.734695:0:31719:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c59a1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d696f expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.734710:0:31719:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c59a1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d696f expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.734722:0:31719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c59a1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d696f expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.734729:0:31719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c59a1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d696f expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.734736:0:31719:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c59a1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d696f expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201687.734747:0:31719:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff90234519d500 x1875064442798848/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201703 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201687.734760:0:31719:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c59a1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d696f expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.734772:0:31719:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201687.734780:0:31719:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201687.734804:0:31719:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 6656 00000080:00010000:1.0:1788201687.734810:0:31719:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) for lock 0x4605c7c97e1c59a1 00000080:00010000:1.0:1788201687.734816:0:31719:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff902352108908) 00000080:00010000:1.0:1788201687.734819:0:31719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c59a1 from it ffffa0b181bebbf8 00010000:00010000:1.0:1788201687.734823:0:31719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c59a1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d696f expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.734836:0:31719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c59a1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d696f expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201687.734860:0:31719:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff902352108908), inode mode 41ed mask 1 00000080:00200000:1.0:1788201687.734910:0:31719:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), flags=25 00000080:00200000:1.0:1788201687.734916:0:31719:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d130b.replay-single, dir=[0x200000007:0x1:0x0](ffff902352108908), intent=0 00000002:00010000:1.0:1788201687.734933:0:31719:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: d130b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788201687.734969:0:31719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c59a8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 31719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.734976:0:31719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c59a8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 31719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.734985:0:31719:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c59a8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 31719 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201687.735015:0:31719:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d500 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31719:1875064442798976:192.168.201.130@tcp:101:touch.0 00000100:00100000:1.0:1788201687.735080:0:31719:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2d80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201687.744502:0:31719:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d500 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31719:1875064442798976:192.168.201.130@tcp:101:touch.0 00010000:00010000:1.0:1788201687.744525:0:31719:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c59a8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d698b expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.744541:0:31719:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c59a8 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d698b expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.744547:0:31719:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c59a8 lrc: 4/1,0 mode: --/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d698b expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.744555:0:31719:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c59a8 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d698b expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.744562:0:31719:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c59a8 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d698b expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.744571:0:31719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c59a8 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d698b expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.744576:0:31719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c59a8 lrc: 5/2,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d698b expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.744581:0:31719:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c59a8 lrc: 4/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d698b expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201687.744590:0:31719:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff90234519d500 x1875064442798976/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 592/584 e 0 to 0 dl 1788201703 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201687.744601:0:31719:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c59a8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d698b expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.744609:0:31719:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 4 res 8589937617/1068 (0 0) 00000080:00200000:1.0:1788201687.744617:0:31719:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201687.744639:0:31719:0:(llite_lib.c:3127:ll_update_inode()) inode=[0x200000bd1:0x42c:0x0], updating i_size 7680 00000080:00200000:1.0:1788201687.744642:0:31719:0:(namei.c:147:ll_iget()) got inode: [0x200000bd1:0x42c:0x0](ffff9023449b6c08): rc = 0 00000080:00010000:1.0:1788201687.744648:0:31719:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42c:0x0](ffff9023449b6c08) for lock 0x4605c7c97e1c59a8 00000080:00010000:1.0:1788201687.744658:0:31719:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42c:0x0](ffff9023449b6c08) 00000080:00010000:1.0:1788201687.744661:0:31719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c59a8 from it ffffa0b181bebac8 00010000:00010000:1.0:1788201687.744664:0:31719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c59a8 lrc: 3/1,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d698b expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.744670:0:31719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c59a8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d698b expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201687.745016:0:31719:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9023449b6c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788201687.745039:0:31719:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x42c:0x0](ffff9023449b6c08), inode mode 41ed mask 3 00000080:00200000:1.0:1788201687.745046:0:31719:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f130b.replay-single, dir=[0x200000bd1:0x42c:0x0](ffff9023449b6c08), file ffff902358031680, open_flags 8941, mode 81b6 opened 0 00000080:00200000:1.0:1788201687.745054:0:31719:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f130b.replay-single, dir=[0x200000bd1:0x42c:0x0](ffff9023449b6c08), intent=open|creat 00000002:00010000:1.0:1788201687.745484:0:31719:0:(mdc_locks.c:1506:mdc_intent_lock()) (name: f130b.replay-single,[0x200000bd1:0x42e:0x0]) in obj [0x200000bd1:0x42c:0x0], intent: open|creat flags 0104502 00010000:00010000:1.0:1788201687.745550:0:31719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x42c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 31719 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.745559:0:31719:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x42c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 31719 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201687.745574:0:31719:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201687.745584:0:31719:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234519d500 x1875064442799232/t0(0) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 752/66408 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201687.745594:0:31719:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 3/0,1 mode: --/CW res: [0x200000bd1:0x42c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 31719 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201687.745615:0:31719:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d500 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31719:1875064442799232:192.168.201.130@tcp:101:touch.0 00000100:00100000:1.0:1788201687.745784:0:31719:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2d80 going to sleep for 11 seconds 00010000:00010000:1.0:1788201687.747197:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201687.753973:0:31719:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d500 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31719:1875064442799232:192.168.201.130@tcp:101:touch.0 00010000:00010000:1.0:1788201687.754003:0:31719:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PW ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 4/0,1 mode: --/CW res: [0x200000bd1:0x42c:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.754025:0:31719:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 4/0,1 mode: --/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.754033:0:31719:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 4/0,1 mode: --/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.754043:0:31719:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 4/0,1 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.754051:0:31719:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 4/0,1 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.754062:0:31719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 5/0,2 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.754067:0:31719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 5/0,2 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.754072:0:31719:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 4/0,1 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201687.754079:0:31719:0:(mdc_locks.c:833:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff90234519d500 x1875064442799232/t30064771079(30064771079) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 752/824 e 0 to 0 dl 1788201703 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201687.754100:0:31719:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90234519d500 x1875064442799232/t30064771079(30064771079) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 752/824 e 0 to 0 dl 1788201703 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201687.754115:0:31719:0:(mdc_locks.c:937:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 216 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 3/0,1 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201687.754126:0:31719:0:(mdc_locks.c:969:mdc_finish_enqueue()) ### DoM lock is returned by: open|creat, size: 0 ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 3/0,1 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201687.754136:0:31719:0:(mdc_locks.c:1357:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 3/0,1 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.754147:0:31719:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff902350c9f400 type 13 mode 2 res 8589937617/1070 (0 0) 00000080:00200000:1.0:1788201687.754153:0:31719:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000bd1:0x42c:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201687.754165:0:31719:0:(namei.c:1628:ll_create_it()) VFS Op:name=f130b.replay-single, dir=[0x200000bd1:0x42c:0x0](ffff9023449b6c08), intent=open|creat 00000080:00200000:1.0:1788201687.754336:0:31719:0:(llite_lib.c:3364:ll_read_inode2()) VFS Op:inode=[0x200000bd1:0x42e:0x0](ffff9023448e5b88) 00000080:00200000:1.0:1788201687.754516:0:31719:0:(file.c:6824:ll_layout_conf()) [0x200000bd1:0x42e:0x0]: layout version change: 4294967294 -> 2 00000080:00010000:1.0:1788201687.754527:0:31719:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000bd1:0x42e:0x0](ffff9023448e5b88) 00000080:00010000:1.0:1788201687.754531:0:31719:0:(llite_internal.h:1988:ll_set_lock_data()) setting l_data to inode [0x200000bd1:0x42e:0x0](ffff9023448e5b88) for lock 0x4605c7c97e1c59af 00000080:00010000:1.0:1788201687.754540:0:31719:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000bd1:0x42e:0x0](ffff9023448e5b88) 00000080:00200008:1.0:1788201687.754552:0:31719:0:(file.c:1090:ll_file_open()) START file f130b.replay-single:[0x200000bd1:0x42e:0x0](ffff9023448e5b88), flags 104501 00000002:00100000:1.0:1788201687.754570:0:31719:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff90234519d500 x1875064442799232/t30064771079(30064771079) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 752/824 e 0 to 0 dl 1788201703 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201687.754615:0:31719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x4605c7c97e1c59af from it ffff9023450e7ba0 00010000:00010000:1.0:1788201687.754620:0:31719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 3/0,1 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201687.754630:0:31719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 2/0,0 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201687.754654:0:31719:0:(file.c:6659:ll_inode_permission()) VFS Op:inode=[0x200000bd1:0x42e:0x0](ffff9023448e5b88), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201687.754686:0:31719:0:(llite_lib.c:2436:ll_setattr_raw()) START file f130b.replay-single:[0x200000bd1:0x42e:0x0](ffff9023448e5b88) current size 0, valid attrs 20070, mode 9023, uid -2118205872, gid -24399, new size 18446744071973566999, atime 1788201687.000000000, mtime 1788201687.000000000, ctime 1788201687.000000000 00000020:00100000:1.0:1788201687.756271:0:31719:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201687.756285:0:31719:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234519e680 x1875064442799488/t0(0) o36->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201687.756309:0:31719:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e680 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31719:1875064442799488:192.168.201.130@tcp:36:touch.0 00000100:00100000:1.0:1788201687.756413:0:31719:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201687.758994:0:2354:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.130@tcp (source: 12345-192.168.201.130@tcp) 00000100:00100000:3.0:1788201687.759051:0:5591:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1788201687.759069:0:5591:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875064458350464 00000100:00100000:3.0:1788201687.759076:0:5591:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875064458350464 00000100:00100000:3.0:1788201687.759137:0:5591:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.130@tcp, seq: 1371 00000100:00100000:3.0:1788201687.759147:0:5591:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff902345191450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:35604eb4-baf9-45be-a20a-4883a2cd8cf9+5:24544:x1875064458350464:12345-192.168.201.130@tcp:104: 00010000:00010000:3.0:1788201687.759165:0:5591:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 2/0,0 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201687.759249:0:5591:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788201687.759256:0:5591:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff902345191450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:35604eb4-baf9-45be-a20a-4883a2cd8cf9+5:24544:x1875064458350464:12345-192.168.201.130@tcp:104: Request processed in 110us (277us total) trans 0 rc 0/0 00000100:00100000:3.0:1788201687.759263:0:5591:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.130@tcp, seq: 1371 00010000:00010000:1.0:1788201687.759381:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201687.759389:0:13086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 2/0,0 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201687.759399:0:13086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff90235265ca00 already unused, calling callback (ffffffffc1a662a0) 00010000:00010000:1.0:1788201687.759425:0:13086:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 2/0,0 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x2 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201687.759438:0:13086:0:(ldlm_request.c:1334:ldlm_cli_convert_req()) ### client-side convert ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 2/0,0 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x5b/0x2 rrc: 2 type: IBT gid 0 flags: 0x420402000000 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201687.759489:0:13086:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 2/0,0 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x59/0x2 rrc: 2 type: IBT gid 0 flags: 0x10420402000000 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201687.759500:0:13086:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 2/0,0 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201687.759513:0:13086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 2/0,0 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201687.759521:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201687.759523:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201687.759525:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201687.759556:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442799616:192.168.201.130@tcp:102:ldlm_bl.0 00000100:00100000:0.0:1788201687.767417:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442799616:192.168.201.130@tcp:102:ldlm_bl.0 00000100:00100000:1.0:1788201687.779985:0:31719:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e680 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31719:1875064442799488:192.168.201.130@tcp:36:touch.0 00000080:00200000:1.0:1788201687.780067:0:31719:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x42e:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201687.780080:0:31719:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:1.0:1788201687.780131:0:31719:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:1.0:1788201687.780171:0:31719:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x42e:0x0] ignore/verify layout 0/0, layout version 2 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1788201687.780202:0:31719:0:(llite_lib.c:2695:ll_setattr_raw()) COMPLETED file f130b.replay-single:[0x200000bd1:0x42e:0x0](ffff9023448e5b88) current size 0, valid attrs 100201f0, mode 9023, uid -2118205872, gid -24399, new size 18446744071973566999, atime 1788201687.000000000, mtime 1788201687.000000000, ctime 1788201687.000000000, rc 0 00000080:00200008:1.0:1788201687.780226:0:31719:0:(file.c:479:ll_file_release()) START file f130b.replay-single:[0x200000bd1:0x42e:0x0](ffff9023448e5b88), flags 104001 00000002:00100000:1.0:1788201687.780275:0:31719:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff90234519d500 x1875064442799232/t30064771079(30064771079) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 752/824 e 0 to 0 dl 1788201703 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201687.780287:0:31719:0:(genops.c:2444:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff902350a11000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201687.780293:0:31719:0:(client.c:807:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff90234519ce00 x1875064442799872/t0(0) o35->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:23/10 lens 392/712 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201687.780312:0:31719:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31719:1875064442799872:192.168.201.130@tcp:35:touch.0 00000100:00100000:1.0:1788201687.780399:0:31719:0:(client.c:2702:ptlrpc_set_wait()) set ffff902342ee2d80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201687.786802:0:31719:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ce00 pname:cluuid:pid:xid:nid:opc:job touch:35604eb4-baf9-45be-a20a-4883a2cd8cf9:31719:1875064442799872:192.168.201.130@tcp:35:touch.0 00000002:00100000:1.0:1788201687.786841:0:31719:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff90234519d500 x1875064442799232/t30064771079(30064771079) o101->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 752/824 e 0 to 0 dl 1788201703 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788201688.771684:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201689.793266:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201690.817117:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201691.841304:0:10725:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 165/165 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1788201691.844168:0:13086:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 165 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201691.844184:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000094550ea9/0x4605c7c97e1ba5e6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x115:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4abb76 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201691.844425:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1788201691.844517:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x22d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.844586:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x22d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.844639:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a82c7147/0x4605c7c97e1ba832 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x12a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac25a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.844663:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x257:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.844679:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x257:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.844698:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cd5a12b/0x4605c7c97e1ba63a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x118:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4abc72 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.845437:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x233:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.845457:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x233:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.845476:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003ab42611/0x4605c7c97e1ba816 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x129:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac206 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.845491:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x255:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.845503:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x255:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.845518:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009209ad20/0x4605c7c97e1ba6fe lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x11f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4abebe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.845536:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x241:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.845549:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x241:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.845563:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cf1a67fe/0x4605c7c97e1ba7fa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x128:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac1b2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.845581:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x253:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.845598:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x253:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.845623:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009415661d/0x4605c7c97e1ba672 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x11a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4abd1a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.845645:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x237:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.845664:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x237:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.845687:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004dcc487d/0x4605c7c97e1ba61e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x117:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4abc1e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.845723:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x231:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.845742:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x231:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.845765:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a4297cd9/0x4605c7c97e1ba602 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x116:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4abbca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.845782:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x22f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.845799:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x22f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.845815:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004fc3f859/0x4605c7c97e1ba5ae lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x113:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4abace expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.845830:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x229:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.845842:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x229:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.845856:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000038165c64/0x4605c7c97e1ba592 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x112:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aba7a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.845870:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x227:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.845897:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x227:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.845913:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005f0b420a/0x4605c7c97e1ba576 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x111:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aba26 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.845928:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x225:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.845939:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x225:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.845953:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000dd42ece4/0x4605c7c97e1ba55a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x110:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab9d2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.845967:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x223:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.845978:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x223:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.845995:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008942b78c/0x4605c7c97e1ba53e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab97e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846026:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x221:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846044:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x221:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846067:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b3572f40/0x4605c7c97e1ba522 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab92a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846092:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x21f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846108:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x21f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846131:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000e8ec782/0x4605c7c97e1ba3d2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x102:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab53a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846155:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x207:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846171:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x207:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846187:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ccb47923/0x4605c7c97e1ba2d6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xf9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab246 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846202:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1f5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846214:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1f5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846228:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004b8407cb/0x4605c7c97e1ba1be lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xef:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aaefe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846242:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1e1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846253:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846267:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003a54fa17/0x4605c7c97e1ba0c2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xe6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aac0a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846281:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1cf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846292:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1cf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846306:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e09ff862/0x4605c7c97e1b9faa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xdc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa8c2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846320:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1bb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846332:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1bb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846346:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005c365dab/0x4605c7c97e1b9eae lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xd3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa5ce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846360:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1a9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846372:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1a9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846386:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008124cfe7/0x4605c7c97e1b9d96 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xc9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa286 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846400:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x195:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846412:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x195:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846426:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bf7b5d8e/0x4605c7c97e1b9c9a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xc0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9f92 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846440:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x183:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846454:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x183:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846476:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008becb8b2/0x4605c7c97e1b9b4a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xb4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9ba2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846499:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x16b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846519:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x16b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846544:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005c770350/0x4605c7c97e1b9a32 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xaa:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a985a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846568:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x157:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846586:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x157:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846610:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007a7a56f2/0x4605c7c97e1b9936 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xa1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9566 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846633:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x145:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846656:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x145:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846680:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b1a80004/0x4605c7c97e1b981e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a921e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846703:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x131:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846723:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x131:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846745:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000910e80e/0x4605c7c97e1ba506 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab8d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846770:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x21d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846789:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x21d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846811:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000053609dd4/0x4605c7c97e1ba4ea lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab882 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846836:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x21b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846855:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x21b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846894:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cae7210/0x4605c7c97e1ba4ce lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab82e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846911:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x219:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846928:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x219:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846943:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000baf25d7b/0x4605c7c97e1ba4b2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab7da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.846957:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x217:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.846969:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x217:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.846983:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005a5aa7f3/0x4605c7c97e1ba496 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x109:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab786 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.847001:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x215:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.847028:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x215:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.847052:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000860030e9/0x4605c7c97e1ba47a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x108:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab732 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.847076:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x213:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.847095:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x213:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.847116:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000666a4d79/0x4605c7c97e1ba45e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x107:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab6de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.847139:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x211:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.847158:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x211:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.847182:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e26bc8d0/0x4605c7c97e1ba442 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x106:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab68a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.847207:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x20f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.847226:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x20f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.847250:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000771e76dd/0x4605c7c97e1ba426 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x105:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab636 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.847273:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x20d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.847291:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x20d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.847316:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000966562e0/0x4605c7c97e1ba40a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x104:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab5e2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.847339:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x20b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.847360:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x20b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.847384:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fd8f24d5/0x4605c7c97e1ba3ee lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x103:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab58e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.847409:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x209:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.847427:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x209:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.847452:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000f6a212f/0x4605c7c97e1ba3b6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x101:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab4e6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.847476:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x205:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.847494:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x205:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.847518:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000350f086b/0x4605c7c97e1ba39a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x100:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab492 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.847542:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x203:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.847559:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x203:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.847582:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000912ae80e/0x4605c7c97e1ba37e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xff:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab43e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.847605:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x201:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.847624:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x201:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.847648:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000026817f33/0x4605c7c97e1ba362 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xfe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab3ea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.847671:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1ff:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.847689:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1ff:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.847712:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002a307d29/0x4605c7c97e1ba346 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xfd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab396 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.847737:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1fd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.847755:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1fd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.847779:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008280ff4d/0x4605c7c97e1ba32a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xfc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab342 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.847801:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1fb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.847825:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1fb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.847849:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f9b233fd/0x4605c7c97e1b96ea lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x8c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8e82 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.847899:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x11b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.847920:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x11b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.847945:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bdc8bc3f/0x4605c7c97e1ba30e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xfb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab2ee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.847969:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1f9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.847990:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1f9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.848013:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f011ad67/0x4605c7c97e1ba2f2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xfa:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab29a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.848396:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1f7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.848417:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1f7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.848443:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000e93fb44/0x4605c7c97e1ba2ba lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xf8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab1f2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.848466:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1f3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.848486:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1f3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.848511:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000056cbf42e/0x4605c7c97e1ba29e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xf7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab19e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.848537:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1f1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.848555:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1f1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.848580:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fe8a1b8a/0x4605c7c97e1ba282 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xf6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab14a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.848604:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1ef:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.848624:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1ef:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.848648:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e208aa43/0x4605c7c97e1ba266 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xf5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab0f6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.848674:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1ed:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.848695:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1ed:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.850101:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d96f0622/0x4605c7c97e1ba24a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xf4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab0a2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.850149:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1eb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.850185:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1eb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.850211:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008602a4f9/0x4605c7c97e1ba22e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xf3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ab04e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.850237:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1e9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.850257:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.850281:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005a867ed6/0x4605c7c97e1ba212 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xf2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aaffa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.850304:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1e7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.850323:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.850347:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bac80f49/0x4605c7c97e1ba1f6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xf1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aafa6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.850370:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1e5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.850389:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.850413:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005d01bf58/0x4605c7c97e1ba1da lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xf0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aaf52 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.850435:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1e3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.850452:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.850476:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f425b3f/0x4605c7c97e1ba1a2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xee:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aaeaa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.850499:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1df:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.850517:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1df:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.850539:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ab7b5872/0x4605c7c97e1ba186 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xed:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aae56 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.850563:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1dd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.850587:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1dd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.850612:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004fe61f38/0x4605c7c97e1ba16a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xec:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aae02 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.850636:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1db:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.850654:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1db:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.850677:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000062d110f4/0x4605c7c97e1ba14e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xeb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aadae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.850701:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1d9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.850730:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1d9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.850755:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000acccd0bc/0x4605c7c97e1ba132 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xea:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aad5a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.850780:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1d7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.850798:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1d7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.850821:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a5c37464/0x4605c7c97e1b95d2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8b3a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.850844:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x107:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.850864:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x107:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.850913:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000059fb2683/0x4605c7c97e1ba116 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xe9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aad06 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.850939:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1d5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.850959:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1d5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.850984:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000002bb26ba/0x4605c7c97e1ba0fa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xe8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aacb2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.851008:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1d3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.851036:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1d3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.851060:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000839286d7/0x4605c7c97e1ba0de lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xe7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aac5e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.851085:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1d1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.851107:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1d1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.851131:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d824af12/0x4605c7c97e1ba0a6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xe5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aabb6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.851155:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1cd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.851173:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1cd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.851197:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b08662c4/0x4605c7c97e1ba08a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xe4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aab62 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.851222:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1cb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.851240:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1cb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.851265:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000dfc768b4/0x4605c7c97e1ba06e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xe3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aab0e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.851288:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1c9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.851306:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1c9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.851330:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e7c36f02/0x4605c7c97e1ba052 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xe2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aaaba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.851355:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1c7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.851373:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1c7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.851397:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000df4cdbe6/0x4605c7c97e1ba036 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xe1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aaa66 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.851421:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1c5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.851439:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1c5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.851461:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004fc0a7ca/0x4605c7c97e1ba01a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xe0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aaa12 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.851484:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1c3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.851500:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1c3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.851523:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fabc291d/0x4605c7c97e1b9ffe lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xdf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa9be expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.851546:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1c1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.851565:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1c1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.851589:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d3a3f5d1/0x4605c7c97e1b9fe2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xde:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa96a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.851613:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1bf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.851631:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1bf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.851653:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002551a452/0x4605c7c97e1b9fc6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xdd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa916 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.851676:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1bd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.851694:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1bd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.851717:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006dda81e5/0x4605c7c97e1b9f8e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xdb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa86e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.851739:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1b9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.851756:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1b9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.851777:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e706f80a/0x4605c7c97e1b9f72 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xda:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa81a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.851797:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1b7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.851813:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1b7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.851832:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b5a8bf82/0x4605c7c97e1b9f56 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xd9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa7c6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.851851:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1b5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.851866:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1b5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.851914:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d891eaf6/0x4605c7c97e1b9f3a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xd8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa772 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.851937:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1b3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.851953:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1b3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.851974:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000da6527da/0x4605c7c97e1b94d6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x79:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8846 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.851996:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xf5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.852027:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xf5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.852051:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003802d1fb/0x4605c7c97e1b9f1e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xd7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa71e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.852075:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1b1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.852094:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1b1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.852116:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000737d9ac3/0x4605c7c97e1b9f02 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xd6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa6ca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.852139:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1af:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.852156:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1af:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.852179:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006da41096/0x4605c7c97e1b9ee6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xd5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa676 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.852202:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1ad:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.852219:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1ad:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.852240:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a972f0e7/0x4605c7c97e1b9eca lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xd4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa622 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.852262:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1ab:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.852278:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1ab:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.852298:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000005a33183/0x4605c7c97e1b9e92 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xd2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa57a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.852318:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1a7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.852332:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1a7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.852352:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001dc67823/0x4605c7c97e1b9e76 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xd1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa526 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.852388:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1a5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.852405:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1a5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.852427:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000078e5b714/0x4605c7c97e1b9e5a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xd0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa4d2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.852448:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1a3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.852470:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1a3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.852494:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000593dfb98/0x4605c7c97e1b9e3e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xcf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa47e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.852518:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x1a1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.852536:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1a1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.852560:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009025ca3b/0x4605c7c97e1b9e22 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa42a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.852584:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x19f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.852601:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x19f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.852625:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000afed9749/0x4605c7c97e1b9e06 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xcd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa3d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.852649:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x19d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.852666:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x19d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.852691:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000095149b2d/0x4605c7c97e1b9dea lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xcc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa382 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.852713:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x19b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.852731:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x19b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.852755:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005ed9071c/0x4605c7c97e1b9dce lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xcb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa32e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.852780:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x199:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.852798:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x199:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.852823:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000966b4fa7/0x4605c7c97e1b9db2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa2da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.852845:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x197:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.852862:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x197:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.852913:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058820789/0x4605c7c97e1b9d7a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xc8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa232 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.852933:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x193:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.852949:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x193:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.852964:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002bc76eed/0x4605c7c97e1b9d5e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xc7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa1de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.852979:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x191:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.852990:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x191:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.853009:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000eabb7b6d/0x4605c7c97e1b9d42 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xc6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa18a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.854236:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x18f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.854263:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x18f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.854284:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000da273d75/0x4605c7c97e1b93be lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a84fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.854300:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xe1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.854313:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xe1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.854328:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d1e3a38d/0x4605c7c97e1b9d26 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xc5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa136 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.854342:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x18d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.854354:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x18d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.854369:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008ae2b79f/0x4605c7c97e1b9d0a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xc4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa0e2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.854382:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x18b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.854393:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x18b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.854408:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007ad0bd20/0x4605c7c97e1b9cee lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xc3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa08e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.854421:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x189:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.854432:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x189:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.854447:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cf96222/0x4605c7c97e1b9cd2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xc2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aa03a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.854461:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x187:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.854475:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x187:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.854489:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a85655e8/0x4605c7c97e1b9cb6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xc1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9fe6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.854504:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x185:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.854515:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x185:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.854529:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006dcff5e4/0x4605c7c97e1b9c7e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xbf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9f3e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.854542:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x181:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.854553:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x181:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.854568:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f44fb731/0x4605c7c97e1b9c62 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xbe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9eea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.854590:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x17f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.854608:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x17f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.854632:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d7b30246/0x4605c7c97e1b9c46 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xbd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9e96 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.854657:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x17d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.854675:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x17d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.854697:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000742c3bac/0x4605c7c97e1b9c2a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xbc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9e42 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.854725:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x17b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.854739:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x17b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.854754:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000eb9fe0f5/0x4605c7c97e1b9c0e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xbb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9dee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.854769:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x179:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.854780:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x179:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.854794:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007d9687f3/0x4605c7c97e1b9bf2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xba:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9d9a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.854808:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x177:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.854821:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x177:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.854835:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000eb832689/0x4605c7c97e1b9bd6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xb9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9d46 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.854849:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x175:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.854860:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x175:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.854888:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000188ebbc9/0x4605c7c97e1b9bba lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xb8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9cf2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.854903:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x173:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.854916:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x173:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.854932:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a86d3aef/0x4605c7c97e1b9b9e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xb7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9c9e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.854946:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x171:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.854957:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x171:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.854971:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d3a46a0d/0x4605c7c97e1b9b82 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xb6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9c4a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.854984:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x16f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.854996:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x16f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855026:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000996e2bda/0x4605c7c97e1b9b66 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xb5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9bf6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855050:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x16d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855067:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x16d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855084:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cf5e4b4e/0x4605c7c97e1b9b2e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xb3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9b4e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855099:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x169:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855114:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x169:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855136:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009d9f5179/0x4605c7c97e1b9b12 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xb2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9afa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855159:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x167:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855175:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x167:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855191:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000dcc0a2ed/0x4605c7c97e1b9af6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xb1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9aa6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855206:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x165:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855216:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x165:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855231:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001a301204/0x4605c7c97e1b9ada lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xb0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9a52 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855244:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x163:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855255:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x163:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855269:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cac79329/0x4605c7c97e1b9abe lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xaf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a99fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855282:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x161:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855293:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x161:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855307:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000371c8493/0x4605c7c97e1b9aa2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xae:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a99aa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855321:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x15f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855332:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x15f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855346:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006a672457/0x4605c7c97e1b9a86 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xad:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9956 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855359:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x15d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855370:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x15d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855384:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000520ea39d/0x4605c7c97e1b9a6a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xac:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9902 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855398:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x15b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855408:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x15b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855422:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000622932d8/0x4605c7c97e1b9a4e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xab:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a98ae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855435:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x159:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855449:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x159:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855470:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a525410b/0x4605c7c97e1b9a16 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9806 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855493:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x155:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855511:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x155:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855533:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008374c11f/0x4605c7c97e1b99fa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a97b2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855557:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x153:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855575:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x153:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855600:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007ee61001/0x4605c7c97e1b99de lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a975e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855623:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x151:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855644:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x151:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855667:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ecdc4ad7/0x4605c7c97e1b99c2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a970a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855690:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x14f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855711:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x14f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855734:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d8322a9e/0x4605c7c97e1b99a6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a96b6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855756:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x14d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855772:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x14d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855795:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ecc17849/0x4605c7c97e1b998a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9662 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855816:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x14b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855833:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x14b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855855:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001cc1d8aa/0x4605c7c97e1b91e2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x5e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7f6a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855908:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xbf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855931:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xbf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.855956:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cee46114/0x4605c7c97e1b996e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xa3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a960e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.855981:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x149:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.855999:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x149:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.856032:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009b64fc1d/0x4605c7c97e1b9952 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xa2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a95ba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.856056:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x147:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.856075:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x147:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.856099:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ffa33635/0x4605c7c97e1b991a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0xa0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9512 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.856124:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x143:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.856144:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x143:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.856165:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000960ddc8d/0x4605c7c97e1b98fe lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a94be expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.856186:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x141:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.856203:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x141:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.856223:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7afab04/0x4605c7c97e1b98e2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a946a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.856244:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x13f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.856260:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x13f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.856284:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2335581/0x4605c7c97e1b98c6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9416 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.856307:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x13d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.856325:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x13d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.856346:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000096e83ca/0x4605c7c97e1b98aa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a93c2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.856367:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x13b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.856386:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x13b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.856406:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000018f05221/0x4605c7c97e1b988e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a936e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.856427:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x139:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.856444:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x139:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.856465:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001987dbcf/0x4605c7c97e1b9872 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a931a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.856485:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x137:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.856502:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x137:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.856522:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001b607bbc/0x4605c7c97e1b9856 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a92c6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.856542:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x135:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.856556:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x135:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.856577:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f07c258c/0x4605c7c97e1b983a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9272 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.856600:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x133:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.856617:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x133:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.856639:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000066218352/0x4605c7c97e1b9802 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a91ca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.856662:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x12f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.856680:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x12f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.856704:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000418dd66e/0x4605c7c97e1b97e6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9176 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.856727:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x12d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.856744:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x12d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.856767:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bd6b4920/0x4605c7c97e1b97ca lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x94:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9122 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.856790:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x12b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.856811:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x12b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.856833:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007a6171f1/0x4605c7c97e1b97ae lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a90ce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.856858:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x129:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.856897:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x129:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.856922:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000912d45f7/0x4605c7c97e1b9792 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a907a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.856946:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x127:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.856963:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x127:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.856987:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000015d69af2/0x4605c7c97e1b90e6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x55:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a7c76 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.857010:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xad:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.858642:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xad:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.858686:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cfdde598/0x4605c7c97e1b9776 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x91:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a9026 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.858736:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x125:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.858755:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x125:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.858777:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a1eca417/0x4605c7c97e1b975a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8fd2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.858799:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x123:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.858814:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x123:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.858834:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d6e61c60/0x4605c7c97e1b973e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8f7e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.858855:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x121:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.858871:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x121:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.858908:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000dad0e96/0x4605c7c97e1b9722 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8f2a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.858928:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x11f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.858948:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x11f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.858968:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000980a1948/0x4605c7c97e1b9706 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8ed6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.858989:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x11d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.859006:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x11d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.859038:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c43a6e5a/0x4605c7c97e1b96ce lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8e2e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.859061:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x119:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.859081:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x119:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.859106:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000228fea03/0x4605c7c97e1b96b2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x8a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8dda expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.859129:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x117:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.859146:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x117:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.859170:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000063768f41/0x4605c7c97e1b9696 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x89:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8d86 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.859192:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x115:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.859217:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x115:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.859239:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081c5deeb/0x4605c7c97e1b967a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x88:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8d32 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.859259:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x113:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.859274:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x113:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.859299:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001adb72e4/0x4605c7c97e1b965e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8cde expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.859320:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x111:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.859336:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x111:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.859357:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000029f82ef2/0x4605c7c97e1b9642 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8c8a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.859377:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x10f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.859393:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x10f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.859413:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006c36470/0x4605c7c97e1b9626 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8c36 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.859433:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x10d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.859449:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x10d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.859469:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d39e1612/0x4605c7c97e1b960a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8be2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.859494:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x10b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.859508:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x10b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.859528:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000423bdd2f/0x4605c7c97e1b95ee lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8b8e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.859555:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x109:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.859575:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x109:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.859599:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006369d676/0x4605c7c97e1b95b6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8ae6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.859623:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x105:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.859642:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x105:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.859665:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fc917e51/0x4605c7c97e1b959a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8a92 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.859711:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x103:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.859728:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x103:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.859750:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000044c9a191/0x4605c7c97e1b8fce lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x4b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a792e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.859858:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x99:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.859902:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x99:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.859926:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004965b770/0x4605c7c97e1b957e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8a3e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.859948:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x101:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.859968:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x101:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.859991:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f350e7dc/0x4605c7c97e1b9562 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a89ea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.860024:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xff:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.860042:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xff:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.860066:0:13086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c7c9fd8d/0x4605c7c97e1b9546 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4a8996 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201691.860089:0:13086:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0xfd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201691.860107:0:13086:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0xfd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201691.860166:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000094550ea9/0x4605c7c97e1ba5e6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x115:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abb76 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860177:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a82c7147/0x4605c7c97e1ba832 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x12a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac25a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860185:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cd5a12b/0x4605c7c97e1ba63a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x118:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abc72 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860193:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003ab42611/0x4605c7c97e1ba816 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x129:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac206 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860201:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009209ad20/0x4605c7c97e1ba6fe lrc: 1/0,0 mode: --/PW res: [0x240000400:0x11f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abebe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860209:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cf1a67fe/0x4605c7c97e1ba7fa lrc: 1/0,0 mode: --/PW res: [0x240000400:0x128:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac1b2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860217:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009415661d/0x4605c7c97e1ba672 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x11a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abd1a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860224:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004dcc487d/0x4605c7c97e1ba61e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x117:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abc1e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860237:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a4297cd9/0x4605c7c97e1ba602 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x116:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abbca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860244:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004fc3f859/0x4605c7c97e1ba5ae lrc: 1/0,0 mode: --/PW res: [0x240000400:0x113:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abace expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860251:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000038165c64/0x4605c7c97e1ba592 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x112:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aba7a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860258:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005f0b420a/0x4605c7c97e1ba576 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x111:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aba26 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860265:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000dd42ece4/0x4605c7c97e1ba55a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x110:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab9d2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860272:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008942b78c/0x4605c7c97e1ba53e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab97e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860279:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b3572f40/0x4605c7c97e1ba522 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab92a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860287:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000e8ec782/0x4605c7c97e1ba3d2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x102:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab53a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860295:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ccb47923/0x4605c7c97e1ba2d6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xf9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab246 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860308:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004b8407cb/0x4605c7c97e1ba1be lrc: 1/0,0 mode: --/PW res: [0x240000400:0xef:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aaefe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860316:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003a54fa17/0x4605c7c97e1ba0c2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xe6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aac0a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860323:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e09ff862/0x4605c7c97e1b9faa lrc: 1/0,0 mode: --/PW res: [0x240000400:0xdc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa8c2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860330:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005c365dab/0x4605c7c97e1b9eae lrc: 1/0,0 mode: --/PW res: [0x240000400:0xd3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa5ce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860337:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008124cfe7/0x4605c7c97e1b9d96 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xc9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa286 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860344:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bf7b5d8e/0x4605c7c97e1b9c9a lrc: 1/0,0 mode: --/PW res: [0x240000400:0xc0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9f92 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860352:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008becb8b2/0x4605c7c97e1b9b4a lrc: 1/0,0 mode: --/PW res: [0x240000400:0xb4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9ba2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860360:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005c770350/0x4605c7c97e1b9a32 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xaa:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a985a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860368:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007a7a56f2/0x4605c7c97e1b9936 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xa1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9566 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860380:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b1a80004/0x4605c7c97e1b981e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a921e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860387:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000910e80e/0x4605c7c97e1ba506 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab8d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860395:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000053609dd4/0x4605c7c97e1ba4ea lrc: 1/0,0 mode: --/PW res: [0x240000400:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab882 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860404:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cae7210/0x4605c7c97e1ba4ce lrc: 1/0,0 mode: --/PW res: [0x240000400:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab82e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860412:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000baf25d7b/0x4605c7c97e1ba4b2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab7da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860419:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005a5aa7f3/0x4605c7c97e1ba496 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x109:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab786 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860427:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000860030e9/0x4605c7c97e1ba47a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x108:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab732 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860435:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000666a4d79/0x4605c7c97e1ba45e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x107:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab6de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860443:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e26bc8d0/0x4605c7c97e1ba442 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x106:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab68a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860456:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000771e76dd/0x4605c7c97e1ba426 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x105:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab636 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860464:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000966562e0/0x4605c7c97e1ba40a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x104:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab5e2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860472:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fd8f24d5/0x4605c7c97e1ba3ee lrc: 1/0,0 mode: --/PW res: [0x240000400:0x103:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab58e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860479:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000f6a212f/0x4605c7c97e1ba3b6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x101:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab4e6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860487:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000350f086b/0x4605c7c97e1ba39a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x100:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab492 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860494:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000912ae80e/0x4605c7c97e1ba37e lrc: 1/0,0 mode: --/PW res: [0x240000400:0xff:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab43e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860502:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000026817f33/0x4605c7c97e1ba362 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xfe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab3ea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860509:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002a307d29/0x4605c7c97e1ba346 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xfd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab396 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860517:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008280ff4d/0x4605c7c97e1ba32a lrc: 1/0,0 mode: --/PW res: [0x240000400:0xfc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab342 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860528:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f9b233fd/0x4605c7c97e1b96ea lrc: 1/0,0 mode: --/PW res: [0x240000400:0x8c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8e82 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860536:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bdc8bc3f/0x4605c7c97e1ba30e lrc: 1/0,0 mode: --/PW res: [0x240000400:0xfb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab2ee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860544:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f011ad67/0x4605c7c97e1ba2f2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xfa:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab29a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860552:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000e93fb44/0x4605c7c97e1ba2ba lrc: 1/0,0 mode: --/PW res: [0x240000400:0xf8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab1f2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860560:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000056cbf42e/0x4605c7c97e1ba29e lrc: 1/0,0 mode: --/PW res: [0x240000400:0xf7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab19e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860567:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fe8a1b8a/0x4605c7c97e1ba282 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xf6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab14a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860574:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e208aa43/0x4605c7c97e1ba266 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xf5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab0f6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860582:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d96f0622/0x4605c7c97e1ba24a lrc: 1/0,0 mode: --/PW res: [0x240000400:0xf4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab0a2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860590:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008602a4f9/0x4605c7c97e1ba22e lrc: 1/0,0 mode: --/PW res: [0x240000400:0xf3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab04e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860603:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005a867ed6/0x4605c7c97e1ba212 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xf2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aaffa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860611:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bac80f49/0x4605c7c97e1ba1f6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xf1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aafa6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860619:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005d01bf58/0x4605c7c97e1ba1da lrc: 1/0,0 mode: --/PW res: [0x240000400:0xf0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aaf52 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860627:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f425b3f/0x4605c7c97e1ba1a2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xee:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aaeaa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860635:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ab7b5872/0x4605c7c97e1ba186 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xed:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aae56 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860642:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004fe61f38/0x4605c7c97e1ba16a lrc: 1/0,0 mode: --/PW res: [0x240000400:0xec:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aae02 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860649:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000062d110f4/0x4605c7c97e1ba14e lrc: 1/0,0 mode: --/PW res: [0x240000400:0xeb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aadae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860656:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000acccd0bc/0x4605c7c97e1ba132 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xea:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aad5a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860663:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a5c37464/0x4605c7c97e1b95d2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8b3a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860674:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000059fb2683/0x4605c7c97e1ba116 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xe9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aad06 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860681:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000002bb26ba/0x4605c7c97e1ba0fa lrc: 1/0,0 mode: --/PW res: [0x240000400:0xe8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aacb2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860689:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000839286d7/0x4605c7c97e1ba0de lrc: 1/0,0 mode: --/PW res: [0x240000400:0xe7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aac5e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860696:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d824af12/0x4605c7c97e1ba0a6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xe5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aabb6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860705:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b08662c4/0x4605c7c97e1ba08a lrc: 1/0,0 mode: --/PW res: [0x240000400:0xe4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aab62 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860712:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000dfc768b4/0x4605c7c97e1ba06e lrc: 1/0,0 mode: --/PW res: [0x240000400:0xe3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aab0e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860720:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e7c36f02/0x4605c7c97e1ba052 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xe2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aaaba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860727:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000df4cdbe6/0x4605c7c97e1ba036 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xe1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aaa66 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860735:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004fc0a7ca/0x4605c7c97e1ba01a lrc: 1/0,0 mode: --/PW res: [0x240000400:0xe0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aaa12 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860747:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fabc291d/0x4605c7c97e1b9ffe lrc: 1/0,0 mode: --/PW res: [0x240000400:0xdf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa9be expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860755:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d3a3f5d1/0x4605c7c97e1b9fe2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xde:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa96a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860762:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002551a452/0x4605c7c97e1b9fc6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xdd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa916 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860769:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006dda81e5/0x4605c7c97e1b9f8e lrc: 1/0,0 mode: --/PW res: [0x240000400:0xdb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa86e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860776:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e706f80a/0x4605c7c97e1b9f72 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xda:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa81a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860784:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b5a8bf82/0x4605c7c97e1b9f56 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xd9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa7c6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860792:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d891eaf6/0x4605c7c97e1b9f3a lrc: 1/0,0 mode: --/PW res: [0x240000400:0xd8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa772 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860799:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000da6527da/0x4605c7c97e1b94d6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x79:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8846 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860806:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003802d1fb/0x4605c7c97e1b9f1e lrc: 1/0,0 mode: --/PW res: [0x240000400:0xd7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa71e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860817:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000737d9ac3/0x4605c7c97e1b9f02 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xd6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa6ca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860824:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006da41096/0x4605c7c97e1b9ee6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xd5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa676 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860832:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a972f0e7/0x4605c7c97e1b9eca lrc: 1/0,0 mode: --/PW res: [0x240000400:0xd4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa622 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860841:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000005a33183/0x4605c7c97e1b9e92 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xd2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa57a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860849:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001dc67823/0x4605c7c97e1b9e76 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xd1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa526 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860857:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000078e5b714/0x4605c7c97e1b9e5a lrc: 1/0,0 mode: --/PW res: [0x240000400:0xd0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa4d2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860866:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000593dfb98/0x4605c7c97e1b9e3e lrc: 1/0,0 mode: --/PW res: [0x240000400:0xcf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa47e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860899:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009025ca3b/0x4605c7c97e1b9e22 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa42a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860908:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000afed9749/0x4605c7c97e1b9e06 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xcd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa3d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860922:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000095149b2d/0x4605c7c97e1b9dea lrc: 1/0,0 mode: --/PW res: [0x240000400:0xcc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa382 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860930:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005ed9071c/0x4605c7c97e1b9dce lrc: 1/0,0 mode: --/PW res: [0x240000400:0xcb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa32e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860938:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000966b4fa7/0x4605c7c97e1b9db2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa2da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860947:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058820789/0x4605c7c97e1b9d7a lrc: 1/0,0 mode: --/PW res: [0x240000400:0xc8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa232 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860954:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002bc76eed/0x4605c7c97e1b9d5e lrc: 1/0,0 mode: --/PW res: [0x240000400:0xc7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa1de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860962:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000eabb7b6d/0x4605c7c97e1b9d42 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xc6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa18a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860970:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000da273d75/0x4605c7c97e1b93be lrc: 1/0,0 mode: --/PW res: [0x240000400:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a84fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860978:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d1e3a38d/0x4605c7c97e1b9d26 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xc5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa136 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860987:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008ae2b79f/0x4605c7c97e1b9d0a lrc: 1/0,0 mode: --/PW res: [0x240000400:0xc4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa0e2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.860999:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007ad0bd20/0x4605c7c97e1b9cee lrc: 1/0,0 mode: --/PW res: [0x240000400:0xc3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa08e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861008:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cf96222/0x4605c7c97e1b9cd2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xc2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa03a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861025:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a85655e8/0x4605c7c97e1b9cb6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xc1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9fe6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861033:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006dcff5e4/0x4605c7c97e1b9c7e lrc: 1/0,0 mode: --/PW res: [0x240000400:0xbf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9f3e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861041:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f44fb731/0x4605c7c97e1b9c62 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xbe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9eea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861050:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d7b30246/0x4605c7c97e1b9c46 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xbd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9e96 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861058:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000742c3bac/0x4605c7c97e1b9c2a lrc: 1/0,0 mode: --/PW res: [0x240000400:0xbc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9e42 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861065:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000eb9fe0f5/0x4605c7c97e1b9c0e lrc: 1/0,0 mode: --/PW res: [0x240000400:0xbb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9dee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861073:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007d9687f3/0x4605c7c97e1b9bf2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xba:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9d9a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861085:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000eb832689/0x4605c7c97e1b9bd6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xb9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9d46 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861093:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000188ebbc9/0x4605c7c97e1b9bba lrc: 1/0,0 mode: --/PW res: [0x240000400:0xb8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9cf2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861100:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a86d3aef/0x4605c7c97e1b9b9e lrc: 1/0,0 mode: --/PW res: [0x240000400:0xb7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9c9e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861108:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d3a46a0d/0x4605c7c97e1b9b82 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xb6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9c4a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861116:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000996e2bda/0x4605c7c97e1b9b66 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xb5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9bf6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861124:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cf5e4b4e/0x4605c7c97e1b9b2e lrc: 1/0,0 mode: --/PW res: [0x240000400:0xb3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9b4e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861131:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009d9f5179/0x4605c7c97e1b9b12 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xb2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9afa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861140:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000dcc0a2ed/0x4605c7c97e1b9af6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xb1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9aa6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861148:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001a301204/0x4605c7c97e1b9ada lrc: 1/0,0 mode: --/PW res: [0x240000400:0xb0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9a52 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861161:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cac79329/0x4605c7c97e1b9abe lrc: 1/0,0 mode: --/PW res: [0x240000400:0xaf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a99fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861169:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000371c8493/0x4605c7c97e1b9aa2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xae:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a99aa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861177:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006a672457/0x4605c7c97e1b9a86 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xad:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9956 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861186:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000520ea39d/0x4605c7c97e1b9a6a lrc: 1/0,0 mode: --/PW res: [0x240000400:0xac:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9902 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861194:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000622932d8/0x4605c7c97e1b9a4e lrc: 1/0,0 mode: --/PW res: [0x240000400:0xab:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a98ae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861203:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a525410b/0x4605c7c97e1b9a16 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9806 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861211:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008374c11f/0x4605c7c97e1b99fa lrc: 1/0,0 mode: --/PW res: [0x240000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a97b2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861220:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007ee61001/0x4605c7c97e1b99de lrc: 1/0,0 mode: --/PW res: [0x240000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a975e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861229:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ecdc4ad7/0x4605c7c97e1b99c2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a970a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861242:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d8322a9e/0x4605c7c97e1b99a6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a96b6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861250:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ecc17849/0x4605c7c97e1b998a lrc: 1/0,0 mode: --/PW res: [0x240000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9662 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861258:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001cc1d8aa/0x4605c7c97e1b91e2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x5e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7f6a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861266:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cee46114/0x4605c7c97e1b996e lrc: 1/0,0 mode: --/PW res: [0x240000400:0xa3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a960e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861274:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009b64fc1d/0x4605c7c97e1b9952 lrc: 1/0,0 mode: --/PW res: [0x240000400:0xa2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a95ba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861282:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ffa33635/0x4605c7c97e1b991a lrc: 1/0,0 mode: --/PW res: [0x240000400:0xa0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9512 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861290:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000960ddc8d/0x4605c7c97e1b98fe lrc: 1/0,0 mode: --/PW res: [0x240000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a94be expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861297:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7afab04/0x4605c7c97e1b98e2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a946a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861305:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2335581/0x4605c7c97e1b98c6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9416 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861317:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000096e83ca/0x4605c7c97e1b98aa lrc: 1/0,0 mode: --/PW res: [0x240000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a93c2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861325:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000018f05221/0x4605c7c97e1b988e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a936e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861334:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001987dbcf/0x4605c7c97e1b9872 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a931a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861342:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001b607bbc/0x4605c7c97e1b9856 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a92c6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861350:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f07c258c/0x4605c7c97e1b983a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9272 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861357:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000066218352/0x4605c7c97e1b9802 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a91ca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861366:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000418dd66e/0x4605c7c97e1b97e6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9176 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861374:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bd6b4920/0x4605c7c97e1b97ca lrc: 1/0,0 mode: --/PW res: [0x240000400:0x94:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9122 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861382:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007a6171f1/0x4605c7c97e1b97ae lrc: 1/0,0 mode: --/PW res: [0x240000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a90ce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861395:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000912d45f7/0x4605c7c97e1b9792 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a907a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861402:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000015d69af2/0x4605c7c97e1b90e6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x55:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7c76 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861410:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cfdde598/0x4605c7c97e1b9776 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x91:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9026 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861418:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a1eca417/0x4605c7c97e1b975a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8fd2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861425:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d6e61c60/0x4605c7c97e1b973e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8f7e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861433:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000dad0e96/0x4605c7c97e1b9722 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8f2a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861441:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000980a1948/0x4605c7c97e1b9706 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8ed6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861449:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c43a6e5a/0x4605c7c97e1b96ce lrc: 1/0,0 mode: --/PW res: [0x240000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8e2e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861458:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000228fea03/0x4605c7c97e1b96b2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x8a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8dda expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861470:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000063768f41/0x4605c7c97e1b9696 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x89:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8d86 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861477:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081c5deeb/0x4605c7c97e1b967a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x88:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8d32 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861484:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001adb72e4/0x4605c7c97e1b965e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8cde expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861491:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000029f82ef2/0x4605c7c97e1b9642 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8c8a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861496:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006c36470/0x4605c7c97e1b9626 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8c36 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861503:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d39e1612/0x4605c7c97e1b960a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8be2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861510:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000423bdd2f/0x4605c7c97e1b95ee lrc: 1/0,0 mode: --/PW res: [0x240000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8b8e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861517:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006369d676/0x4605c7c97e1b95b6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8ae6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861524:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fc917e51/0x4605c7c97e1b959a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8a92 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861537:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000044c9a191/0x4605c7c97e1b8fce lrc: 1/0,0 mode: --/PW res: [0x240000400:0x4b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a792e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861544:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004965b770/0x4605c7c97e1b957e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8a3e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861550:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f350e7dc/0x4605c7c97e1b9562 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a89ea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861557:0:13086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c7c9fd8d/0x4605c7c97e1b9546 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8996 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861563:0:13086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 165 locks packed 00010000:00010000:1.0:1788201691.861608:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000094550ea9/0x4605c7c97e1ba5e6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x115:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abb76 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861635:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a82c7147/0x4605c7c97e1ba832 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x12a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac25a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861649:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cd5a12b/0x4605c7c97e1ba63a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x118:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abc72 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861662:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003ab42611/0x4605c7c97e1ba816 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x129:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac206 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861675:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009209ad20/0x4605c7c97e1ba6fe lrc: 0/0,0 mode: --/PW res: [0x240000400:0x11f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abebe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861691:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cf1a67fe/0x4605c7c97e1ba7fa lrc: 0/0,0 mode: --/PW res: [0x240000400:0x128:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac1b2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861703:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009415661d/0x4605c7c97e1ba672 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x11a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abd1a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861715:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004dcc487d/0x4605c7c97e1ba61e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x117:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abc1e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861728:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a4297cd9/0x4605c7c97e1ba602 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x116:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abbca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1788201691.861732:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442800000:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201691.861744:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004fc3f859/0x4605c7c97e1ba5ae lrc: 0/0,0 mode: --/PW res: [0x240000400:0x113:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abace expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861756:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000038165c64/0x4605c7c97e1ba592 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x112:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aba7a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861769:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005f0b420a/0x4605c7c97e1ba576 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x111:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aba26 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861782:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000dd42ece4/0x4605c7c97e1ba55a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x110:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab9d2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861800:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008942b78c/0x4605c7c97e1ba53e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x10f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab97e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861813:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b3572f40/0x4605c7c97e1ba522 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x10e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab92a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861825:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000e8ec782/0x4605c7c97e1ba3d2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x102:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab53a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861836:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ccb47923/0x4605c7c97e1ba2d6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xf9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab246 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861848:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004b8407cb/0x4605c7c97e1ba1be lrc: 0/0,0 mode: --/PW res: [0x240000400:0xef:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aaefe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861861:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003a54fa17/0x4605c7c97e1ba0c2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xe6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aac0a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861912:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e09ff862/0x4605c7c97e1b9faa lrc: 0/0,0 mode: --/PW res: [0x240000400:0xdc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa8c2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861928:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005c365dab/0x4605c7c97e1b9eae lrc: 0/0,0 mode: --/PW res: [0x240000400:0xd3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa5ce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861947:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008124cfe7/0x4605c7c97e1b9d96 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xc9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa286 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861961:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bf7b5d8e/0x4605c7c97e1b9c9a lrc: 0/0,0 mode: --/PW res: [0x240000400:0xc0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9f92 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861974:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008becb8b2/0x4605c7c97e1b9b4a lrc: 0/0,0 mode: --/PW res: [0x240000400:0xb4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9ba2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.861987:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005c770350/0x4605c7c97e1b9a32 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xaa:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a985a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862000:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007a7a56f2/0x4605c7c97e1b9936 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xa1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9566 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862013:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b1a80004/0x4605c7c97e1b981e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x97:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a921e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862725:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000910e80e/0x4605c7c97e1ba506 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x10d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab8d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862737:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000053609dd4/0x4605c7c97e1ba4ea lrc: 0/0,0 mode: --/PW res: [0x240000400:0x10c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab882 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862753:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cae7210/0x4605c7c97e1ba4ce lrc: 0/0,0 mode: --/PW res: [0x240000400:0x10b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab82e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862763:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000baf25d7b/0x4605c7c97e1ba4b2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x10a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab7da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862775:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005a5aa7f3/0x4605c7c97e1ba496 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x109:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab786 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862786:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000860030e9/0x4605c7c97e1ba47a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x108:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab732 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862797:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000666a4d79/0x4605c7c97e1ba45e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x107:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab6de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862809:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e26bc8d0/0x4605c7c97e1ba442 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x106:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab68a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862822:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000771e76dd/0x4605c7c97e1ba426 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x105:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab636 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862834:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000966562e0/0x4605c7c97e1ba40a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x104:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab5e2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862849:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fd8f24d5/0x4605c7c97e1ba3ee lrc: 0/0,0 mode: --/PW res: [0x240000400:0x103:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab58e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862860:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000f6a212f/0x4605c7c97e1ba3b6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x101:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab4e6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862871:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000350f086b/0x4605c7c97e1ba39a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x100:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab492 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862908:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000912ae80e/0x4605c7c97e1ba37e lrc: 0/0,0 mode: --/PW res: [0x240000400:0xff:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab43e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862919:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000026817f33/0x4605c7c97e1ba362 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xfe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab3ea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862931:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002a307d29/0x4605c7c97e1ba346 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xfd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab396 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862942:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008280ff4d/0x4605c7c97e1ba32a lrc: 0/0,0 mode: --/PW res: [0x240000400:0xfc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab342 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862952:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f9b233fd/0x4605c7c97e1b96ea lrc: 0/0,0 mode: --/PW res: [0x240000400:0x8c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8e82 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862968:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bdc8bc3f/0x4605c7c97e1ba30e lrc: 0/0,0 mode: --/PW res: [0x240000400:0xfb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab2ee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862978:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f011ad67/0x4605c7c97e1ba2f2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xfa:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab29a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862989:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000e93fb44/0x4605c7c97e1ba2ba lrc: 0/0,0 mode: --/PW res: [0x240000400:0xf8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab1f2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.862999:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000056cbf42e/0x4605c7c97e1ba29e lrc: 0/0,0 mode: --/PW res: [0x240000400:0xf7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab19e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863010:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fe8a1b8a/0x4605c7c97e1ba282 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xf6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab14a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863031:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e208aa43/0x4605c7c97e1ba266 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xf5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab0f6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863044:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d96f0622/0x4605c7c97e1ba24a lrc: 0/0,0 mode: --/PW res: [0x240000400:0xf4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab0a2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863055:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008602a4f9/0x4605c7c97e1ba22e lrc: 0/0,0 mode: --/PW res: [0x240000400:0xf3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ab04e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863072:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005a867ed6/0x4605c7c97e1ba212 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xf2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aaffa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863083:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bac80f49/0x4605c7c97e1ba1f6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xf1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aafa6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863094:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005d01bf58/0x4605c7c97e1ba1da lrc: 0/0,0 mode: --/PW res: [0x240000400:0xf0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aaf52 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863106:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003f425b3f/0x4605c7c97e1ba1a2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xee:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aaeaa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863118:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ab7b5872/0x4605c7c97e1ba186 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xed:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aae56 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863129:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004fe61f38/0x4605c7c97e1ba16a lrc: 0/0,0 mode: --/PW res: [0x240000400:0xec:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aae02 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863140:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000062d110f4/0x4605c7c97e1ba14e lrc: 0/0,0 mode: --/PW res: [0x240000400:0xeb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aadae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863152:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000acccd0bc/0x4605c7c97e1ba132 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xea:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aad5a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863167:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a5c37464/0x4605c7c97e1b95d2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x82:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8b3a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863179:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000059fb2683/0x4605c7c97e1ba116 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xe9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aad06 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863189:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000002bb26ba/0x4605c7c97e1ba0fa lrc: 0/0,0 mode: --/PW res: [0x240000400:0xe8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aacb2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863200:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000839286d7/0x4605c7c97e1ba0de lrc: 0/0,0 mode: --/PW res: [0x240000400:0xe7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aac5e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863213:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d824af12/0x4605c7c97e1ba0a6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xe5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aabb6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863224:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b08662c4/0x4605c7c97e1ba08a lrc: 0/0,0 mode: --/PW res: [0x240000400:0xe4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aab62 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863236:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000dfc768b4/0x4605c7c97e1ba06e lrc: 0/0,0 mode: --/PW res: [0x240000400:0xe3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aab0e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863247:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e7c36f02/0x4605c7c97e1ba052 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xe2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aaaba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863263:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000df4cdbe6/0x4605c7c97e1ba036 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xe1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aaa66 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863276:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004fc0a7ca/0x4605c7c97e1ba01a lrc: 0/0,0 mode: --/PW res: [0x240000400:0xe0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aaa12 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863289:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fabc291d/0x4605c7c97e1b9ffe lrc: 0/0,0 mode: --/PW res: [0x240000400:0xdf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa9be expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863299:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d3a3f5d1/0x4605c7c97e1b9fe2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xde:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa96a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863310:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002551a452/0x4605c7c97e1b9fc6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xdd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa916 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863321:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006dda81e5/0x4605c7c97e1b9f8e lrc: 0/0,0 mode: --/PW res: [0x240000400:0xdb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa86e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863333:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e706f80a/0x4605c7c97e1b9f72 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xda:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa81a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863344:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b5a8bf82/0x4605c7c97e1b9f56 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xd9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa7c6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863361:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d891eaf6/0x4605c7c97e1b9f3a lrc: 0/0,0 mode: --/PW res: [0x240000400:0xd8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa772 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863372:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000da6527da/0x4605c7c97e1b94d6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x79:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8846 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863384:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003802d1fb/0x4605c7c97e1b9f1e lrc: 0/0,0 mode: --/PW res: [0x240000400:0xd7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa71e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863393:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000737d9ac3/0x4605c7c97e1b9f02 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xd6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa6ca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863404:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006da41096/0x4605c7c97e1b9ee6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xd5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa676 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863414:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a972f0e7/0x4605c7c97e1b9eca lrc: 0/0,0 mode: --/PW res: [0x240000400:0xd4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa622 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863425:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000005a33183/0x4605c7c97e1b9e92 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xd2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa57a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863435:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001dc67823/0x4605c7c97e1b9e76 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xd1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa526 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863450:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000078e5b714/0x4605c7c97e1b9e5a lrc: 0/0,0 mode: --/PW res: [0x240000400:0xd0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa4d2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863461:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000593dfb98/0x4605c7c97e1b9e3e lrc: 0/0,0 mode: --/PW res: [0x240000400:0xcf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa47e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863472:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009025ca3b/0x4605c7c97e1b9e22 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa42a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863482:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000afed9749/0x4605c7c97e1b9e06 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xcd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa3d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863493:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000095149b2d/0x4605c7c97e1b9dea lrc: 0/0,0 mode: --/PW res: [0x240000400:0xcc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa382 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863503:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005ed9071c/0x4605c7c97e1b9dce lrc: 0/0,0 mode: --/PW res: [0x240000400:0xcb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa32e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863514:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000966b4fa7/0x4605c7c97e1b9db2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa2da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863525:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058820789/0x4605c7c97e1b9d7a lrc: 0/0,0 mode: --/PW res: [0x240000400:0xc8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa232 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863541:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002bc76eed/0x4605c7c97e1b9d5e lrc: 0/0,0 mode: --/PW res: [0x240000400:0xc7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa1de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863553:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000eabb7b6d/0x4605c7c97e1b9d42 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xc6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa18a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863566:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000da273d75/0x4605c7c97e1b93be lrc: 0/0,0 mode: --/PW res: [0x240000400:0x6f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a84fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863577:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d1e3a38d/0x4605c7c97e1b9d26 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xc5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa136 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863589:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008ae2b79f/0x4605c7c97e1b9d0a lrc: 0/0,0 mode: --/PW res: [0x240000400:0xc4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa0e2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863602:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007ad0bd20/0x4605c7c97e1b9cee lrc: 0/0,0 mode: --/PW res: [0x240000400:0xc3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa08e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863612:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cf96222/0x4605c7c97e1b9cd2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xc2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aa03a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863622:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a85655e8/0x4605c7c97e1b9cb6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xc1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9fe6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863636:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006dcff5e4/0x4605c7c97e1b9c7e lrc: 0/0,0 mode: --/PW res: [0x240000400:0xbf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9f3e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863645:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f44fb731/0x4605c7c97e1b9c62 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xbe:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9eea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863655:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d7b30246/0x4605c7c97e1b9c46 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xbd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9e96 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863665:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000742c3bac/0x4605c7c97e1b9c2a lrc: 0/0,0 mode: --/PW res: [0x240000400:0xbc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9e42 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863675:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000eb9fe0f5/0x4605c7c97e1b9c0e lrc: 0/0,0 mode: --/PW res: [0x240000400:0xbb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9dee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863684:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007d9687f3/0x4605c7c97e1b9bf2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xba:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9d9a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863695:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000eb832689/0x4605c7c97e1b9bd6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xb9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9d46 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863705:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000188ebbc9/0x4605c7c97e1b9bba lrc: 0/0,0 mode: --/PW res: [0x240000400:0xb8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9cf2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863720:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a86d3aef/0x4605c7c97e1b9b9e lrc: 0/0,0 mode: --/PW res: [0x240000400:0xb7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9c9e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863731:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d3a46a0d/0x4605c7c97e1b9b82 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xb6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9c4a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863744:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000996e2bda/0x4605c7c97e1b9b66 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xb5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9bf6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863755:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cf5e4b4e/0x4605c7c97e1b9b2e lrc: 0/0,0 mode: --/PW res: [0x240000400:0xb3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9b4e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863766:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009d9f5179/0x4605c7c97e1b9b12 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xb2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9afa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863778:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000dcc0a2ed/0x4605c7c97e1b9af6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xb1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9aa6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863790:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001a301204/0x4605c7c97e1b9ada lrc: 0/0,0 mode: --/PW res: [0x240000400:0xb0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9a52 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863802:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cac79329/0x4605c7c97e1b9abe lrc: 0/0,0 mode: --/PW res: [0x240000400:0xaf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a99fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863818:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000371c8493/0x4605c7c97e1b9aa2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xae:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a99aa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863829:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006a672457/0x4605c7c97e1b9a86 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xad:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9956 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863839:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000520ea39d/0x4605c7c97e1b9a6a lrc: 0/0,0 mode: --/PW res: [0x240000400:0xac:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9902 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863848:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000622932d8/0x4605c7c97e1b9a4e lrc: 0/0,0 mode: --/PW res: [0x240000400:0xab:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a98ae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863858:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a525410b/0x4605c7c97e1b9a16 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xa9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9806 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863867:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008374c11f/0x4605c7c97e1b99fa lrc: 0/0,0 mode: --/PW res: [0x240000400:0xa8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a97b2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863896:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007ee61001/0x4605c7c97e1b99de lrc: 0/0,0 mode: --/PW res: [0x240000400:0xa7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a975e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863908:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ecdc4ad7/0x4605c7c97e1b99c2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xa6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a970a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863923:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d8322a9e/0x4605c7c97e1b99a6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xa5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a96b6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863933:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ecc17849/0x4605c7c97e1b998a lrc: 0/0,0 mode: --/PW res: [0x240000400:0xa4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9662 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863942:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001cc1d8aa/0x4605c7c97e1b91e2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x5e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7f6a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863952:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cee46114/0x4605c7c97e1b996e lrc: 0/0,0 mode: --/PW res: [0x240000400:0xa3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a960e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863962:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009b64fc1d/0x4605c7c97e1b9952 lrc: 0/0,0 mode: --/PW res: [0x240000400:0xa2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a95ba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863973:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ffa33635/0x4605c7c97e1b991a lrc: 0/0,0 mode: --/PW res: [0x240000400:0xa0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9512 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863983:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000960ddc8d/0x4605c7c97e1b98fe lrc: 0/0,0 mode: --/PW res: [0x240000400:0x9f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a94be expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.863991:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7afab04/0x4605c7c97e1b98e2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x9e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a946a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864006:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2335581/0x4605c7c97e1b98c6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x9d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9416 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864026:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000096e83ca/0x4605c7c97e1b98aa lrc: 0/0,0 mode: --/PW res: [0x240000400:0x9c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a93c2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864037:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000018f05221/0x4605c7c97e1b988e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x9b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a936e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864047:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001987dbcf/0x4605c7c97e1b9872 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x9a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a931a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864057:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001b607bbc/0x4605c7c97e1b9856 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x99:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a92c6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864065:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f07c258c/0x4605c7c97e1b983a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x98:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9272 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864075:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000066218352/0x4605c7c97e1b9802 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a91ca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864085:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000418dd66e/0x4605c7c97e1b97e6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x95:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9176 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864100:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bd6b4920/0x4605c7c97e1b97ca lrc: 0/0,0 mode: --/PW res: [0x240000400:0x94:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9122 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864109:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007a6171f1/0x4605c7c97e1b97ae lrc: 0/0,0 mode: --/PW res: [0x240000400:0x93:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a90ce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864118:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000912d45f7/0x4605c7c97e1b9792 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x92:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a907a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864129:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000015d69af2/0x4605c7c97e1b90e6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x55:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a7c76 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864139:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cfdde598/0x4605c7c97e1b9776 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x91:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a9026 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864148:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a1eca417/0x4605c7c97e1b975a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x90:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8fd2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864159:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d6e61c60/0x4605c7c97e1b973e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x8f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8f7e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864168:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000dad0e96/0x4605c7c97e1b9722 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x8e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8f2a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864183:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000980a1948/0x4605c7c97e1b9706 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x8d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8ed6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864194:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c43a6e5a/0x4605c7c97e1b96ce lrc: 0/0,0 mode: --/PW res: [0x240000400:0x8b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8e2e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864204:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000228fea03/0x4605c7c97e1b96b2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x8a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8dda expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864214:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000063768f41/0x4605c7c97e1b9696 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x89:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8d86 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864224:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000081c5deeb/0x4605c7c97e1b967a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x88:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8d32 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864235:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001adb72e4/0x4605c7c97e1b965e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x87:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8cde expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864249:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000029f82ef2/0x4605c7c97e1b9642 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x86:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8c8a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864260:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006c36470/0x4605c7c97e1b9626 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x85:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8c36 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864278:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d39e1612/0x4605c7c97e1b960a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x84:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8be2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864288:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000423bdd2f/0x4605c7c97e1b95ee lrc: 0/0,0 mode: --/PW res: [0x240000400:0x83:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8b8e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864300:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006369d676/0x4605c7c97e1b95b6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x81:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8ae6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864313:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fc917e51/0x4605c7c97e1b959a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x80:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8a92 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864324:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000044c9a191/0x4605c7c97e1b8fce lrc: 0/0,0 mode: --/PW res: [0x240000400:0x4b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a792e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864336:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004965b770/0x4605c7c97e1b957e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x7f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8a3e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864347:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f350e7dc/0x4605c7c97e1b9562 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x7e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a89ea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864357:0:13086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c7c9fd8d/0x4605c7c97e1b9546 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x7d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4a8996 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201691.864379:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201691.864382:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201691.864384:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201692.034390:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442800128:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201692.034524:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442800384:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201692.034562:0:2362:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201692.034569:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442800256:192.168.201.130@tcp:400:kworker.0 00010000:00010000:0.0:1788201692.865135:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201693.889144:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201694.913145:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201695.937102:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201696.961072:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201697.153136:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442800640:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201697.153136:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442800512:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201697.153230:0:2360:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442800896:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201697.153236:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442800768:192.168.201.130@tcp:400:kworker.0 00010000:00010000:1.0:1788201697.985021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201699.009038:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201700.033019:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201701.057034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201702.081217:0:10725:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 217/217 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1788201702.081254:0:5598:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 217 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201702.081262:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bed84690/0x4605c7c97e1bade2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x15e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad36a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201702.081301:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1788201702.081315:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2bf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.081365:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2bf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.081404:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000276613e8/0x4605c7c97e1bb82a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1bc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af242 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.081420:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x37b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.081436:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x37b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.081458:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004e17030d/0x4605c7c97e1baa0e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x13b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac7ee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.081481:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x279:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.081497:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x279:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.081512:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cbad7b7/0x4605c7c97e1bbb3a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1d8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4afb72 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.081526:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3b3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.081538:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3b3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.081559:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000ed51db4/0x4605c7c97e1bac3e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x14f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ace7e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.081581:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2a1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.081600:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2a1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.081629:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005bceb709/0x4605c7c97e1ba7a6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x125:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac0b6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.081652:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x24d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.081669:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x24d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.081684:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005aec1927/0x4605c7c97e1ba886 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x12d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac356 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.081698:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x25d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.081709:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x25d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.081722:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fc124b78/0x4605c7c97e1bad72 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x15a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad21a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.081743:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2b7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.081758:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2b7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.081778:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000022a4dde9/0x4605c7c97e1bb33e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x18f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae37e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.081800:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x321:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.081816:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x321:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.081831:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bfa2f08d/0x4605c7c97e1bbd32 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1ea:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4b015a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.081867:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3d7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.081902:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3d7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.081920:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000013c8f5e9/0x4605c7c97e1bb2ea lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x18c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae282 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.081943:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x31b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.081968:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x31b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.081990:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005246a2b4/0x4605c7c97e1bb2b2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x18a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae1da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.082012:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x317:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.082032:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x317:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.082055:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000024f5da4d/0x4605c7c97e1bbc36 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1e1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4afe66 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.082077:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3c5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.082096:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3c5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.082118:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005661ba59/0x4605c7c97e1bb27a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x188:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae132 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.082140:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x313:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.082159:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x313:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.082191:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b5f9e4b1/0x4605c7c97e1bbbc6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1dd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4afd16 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.082215:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3bd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.082236:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3bd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.082260:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009e0a5d36/0x4605c7c97e1bb9b2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af6da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.082283:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x397:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.082301:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x397:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.082325:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000368e7f31/0x4605c7c97e1bb0f2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x17a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4adc9a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.082347:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2f7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.082365:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2f7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.082387:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c99af6d6/0x4605c7c97e1bb7ba lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1b8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af0f2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.082410:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x373:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.082429:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x373:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.082452:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007e84d9bc/0x4605c7c97e1babb2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x14a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4accda expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.082475:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x297:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.082493:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x297:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.082516:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001c7d459f/0x4605c7c97e1bab5e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x147:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4acbde expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.082538:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x291:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.082556:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x291:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.082579:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007814fa75/0x4605c7c97e1ba656 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x119:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4abcc6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.082602:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x235:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.082621:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x235:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.082644:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e6acb7a1/0x4605c7c97e1ba76e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x123:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac00e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.082668:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x249:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.082689:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x249:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.082711:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fd4f04bc/0x4605c7c97e1ba8da lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x130:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac452 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.082734:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x263:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.082751:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x263:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.082775:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000de1f3c5/0x4605c7c97e1bae6e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x163:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad50e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.082797:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2c9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.082815:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2c9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.082851:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c5544a0c/0x4605c7c97e1ba736 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x121:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4abf66 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.082891:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x245:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.082911:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x245:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.082934:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ac29708b/0x4605c7c97e1bb41e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x197:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae61e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.082957:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x331:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.082976:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x331:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.082998:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b6fbaa07/0x4605c7c97e1bb392 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x192:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae47a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.083020:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x327:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.083038:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x327:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.083060:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000104ead1a/0x4605c7c97e1ba84e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x12b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac2ae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.083084:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x259:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.083101:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x259:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.083123:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000efb3dfe1/0x4605c7c97e1bb322 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x18e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae32a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.083145:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x31f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.083166:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x31f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.083188:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001c5bb60f/0x4605c7c97e1bad1e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x157:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad11e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.083209:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2b1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.083226:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2b1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.083247:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000d743726/0x4605c7c97e1ba71a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x120:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4abf12 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.083269:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x243:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.083286:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x243:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.083307:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e97d125/0x4605c7c97e1bbc6e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1e3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aff0e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.083330:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3c9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.083345:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3c9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.083366:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000032bcdd95/0x4605c7c97e1ba7de lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x127:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac15e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.083388:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x251:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.083405:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x251:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.083428:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000040396122/0x4605c7c97e1bacca lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x154:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad022 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.083449:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2ab:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.083470:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2ab:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.083493:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000854cfd71/0x4605c7c97e1bbb1e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1d7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4afb1e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.083515:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3b1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.083532:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3b1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.083554:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005340f48b/0x4605c7c97e1bbb02 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1d6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4afaca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.083576:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3af:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.083596:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3af:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.083619:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000062656217/0x4605c7c97e1bbae6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1d5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4afa76 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.083641:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3ad:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.083658:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3ad:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.083681:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f1570ec1/0x4605c7c97e1bbaca lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1d4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4afa22 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.083704:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3ab:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.083721:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3ab:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.083744:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006a7b1128/0x4605c7c97e1bbaae lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1d3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af9ce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.083766:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3a9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.083783:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3a9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.083805:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000235e5f70/0x4605c7c97e1bba92 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1d2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af97a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.083826:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3a7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.083858:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3a7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.083896:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000066b8ebd6/0x4605c7c97e1bba76 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1d1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af926 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.083920:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3a5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.083936:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3a5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.083957:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007e635a94/0x4605c7c97e1bba5a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af8d2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.083979:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3a3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.083995:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3a3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.084018:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009055f3c2/0x4605c7c97e1bba3e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1cf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af87e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.084040:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3a1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.084060:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3a1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.084083:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000567b46ee/0x4605c7c97e1bb10e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x17b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4adcee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.084106:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2f9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.084124:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2f9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.084146:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005b2da7a3/0x4605c7c97e1ba966 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x135:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac5f6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.084168:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x26d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.084186:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x26d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.084208:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e46c2198/0x4605c7c97e1bb89a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1c0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af392 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.084231:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x383:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.084246:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x383:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.084268:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b0abece8/0x4605c7c97e1bb862 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1be:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af2ea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.084290:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x37f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.084309:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x37f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.084331:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f59864fa/0x4605c7c97e1bb7f2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1ba:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af19a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.084356:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x377:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.084375:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x377:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.084399:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000475942c0/0x4605c7c97e1bb04a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x174:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4adaa2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.084422:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2eb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.084440:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2eb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.084466:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f2835f0f/0x4605c7c97e1bb72e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1b3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aef4e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.084489:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x369:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.084509:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x369:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.084533:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c8673266/0x4605c7c97e1bb012 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x172:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad9fa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.084556:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2e7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.084574:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2e7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.084597:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000827b78a6/0x4605c7c97e1ba78a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x124:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac062 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.084620:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x24b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.084637:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x24b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.084660:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009819f599/0x4605c7c97e1bafda lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x170:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad952 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.084681:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2e3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.084699:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2e3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.084722:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002ce3e984/0x4605c7c97e1bab7a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x148:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4acc32 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.084745:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x293:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.084763:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x293:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.084786:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000aa6e73d1/0x4605c7c97e1bafbe lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x16f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad8fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.084808:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2e1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.084825:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2e1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.084862:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000023f07ef9/0x4605c7c97e1ba912 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x132:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac4fa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.084905:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x267:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.084923:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x267:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.084946:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003fd95c5c/0x4605c7c97e1bafa2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x16e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad8aa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.084968:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2df:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.084985:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2df:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085005:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bf9dcc7a/0x4605c7c97e1baf86 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x16d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad856 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085026:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2dd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085042:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2dd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085063:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007408a795/0x4605c7c97e1baf6a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x16c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad802 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085085:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2db:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085100:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2db:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085121:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009f8d40b4/0x4605c7c97e1bab42 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x146:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4acb8a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085142:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x28f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085159:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x28f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085180:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004197033e/0x4605c7c97e1baf4e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x16b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad7ae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085199:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2d9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085214:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2d9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085234:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ae1be017/0x4605c7c97e1ba8f6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x131:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac4a6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085253:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x265:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085267:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x265:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085284:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000367f034/0x4605c7c97e1baf32 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x16a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad75a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085302:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2d7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085316:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2d7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085334:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7408653/0x4605c7c97e1bab26 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x145:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4acb36 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085355:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x28d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085371:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x28d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085392:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000137bbb8d/0x4605c7c97e1baf16 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x169:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad706 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085415:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2d5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085431:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2d5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085452:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000025757b46/0x4605c7c97e1baefa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x168:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad6b2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085474:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2d3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085491:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2d3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085512:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ee55191d/0x4605c7c97e1baea6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x165:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad5b6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085533:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2cd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085548:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2cd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085564:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000098620651/0x4605c7c97e1bb4e2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x19e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae86a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085578:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x33f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085588:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x33f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085604:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006e6fa8b/0x4605c7c97e1bb4c6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x19d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae816 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085625:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x33d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085641:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x33d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085659:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006f703151/0x4605c7c97e1bbe66 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1f5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4b04f6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085673:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3ed:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085686:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3ed:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085699:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000100df4ce/0x4605c7c97e1bb48e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x19b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae76e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085712:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x339:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085724:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x339:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085737:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000842865e9/0x4605c7c97e1bbe4a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1f4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4b04a2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085749:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3eb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085759:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3eb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085772:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000074b50444/0x4605c7c97e1bbe2e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1f3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4b044e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085784:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3e9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085795:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3e9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085807:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006e3de7cc/0x4605c7c97e1bb472 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x19a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae71a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085819:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x337:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085829:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x337:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085856:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000021db2522/0x4605c7c97e1bb456 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x199:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae6c6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085871:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x335:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085898:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x335:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085913:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000048a5c5a4/0x4605c7c97e1bb43a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x198:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae672 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085927:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x333:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085939:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x333:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085951:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005ecb8ec4/0x4605c7c97e1bb402 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x196:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae5ca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085963:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x32f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.085973:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x32f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.085986:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000671e6492/0x4605c7c97e1bb3e6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x195:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae576 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.085998:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x32d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086009:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x32d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086022:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e859d16e/0x4605c7c97e1bb3ca lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x194:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae522 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086042:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x32b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086058:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x32b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086080:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a3abdab2/0x4605c7c97e1bb3ae lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x193:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae4ce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086102:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x329:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086117:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x329:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086133:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000023f36339/0x4605c7c97e1bbe12 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1f2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4b03fa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086147:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3e7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086157:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3e7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086170:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000134a3795/0x4605c7c97e1bbdf6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1f1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4b03a6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086184:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3e5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086198:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3e5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086221:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000a600c03/0x4605c7c97e1bbdda lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1f0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4b0352 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086240:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3e3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086253:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3e3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086267:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002e8b241c/0x4605c7c97e1bbdbe lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1ef:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4b02fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086280:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3e1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086290:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3e1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086303:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000013d10d82/0x4605c7c97e1bb376 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x191:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae426 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086315:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x325:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086326:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x325:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086338:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000062a5f2b4/0x4605c7c97e1bbda2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1ee:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4b02aa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086351:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3df:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086360:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3df:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086373:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ac503f81/0x4605c7c97e1bbd86 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1ed:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4b0256 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086385:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3dd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086396:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3dd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086409:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000063f19c32/0x4605c7c97e1bad3a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x158:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad172 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086421:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2b3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086431:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2b3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086444:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000be7eeb64/0x4605c7c97e1bbd6a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1ec:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4b0202 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086456:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3db:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086466:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3db:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086479:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ff767e23/0x4605c7c97e1bbd4e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1eb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4b01ae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086491:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3d9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086500:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3d9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086513:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c26ae6a4/0x4605c7c97e1ba9f2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x13a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac79a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086525:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x277:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086535:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x277:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086548:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003db3ea37/0x4605c7c97e1bb306 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x18d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae2d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086560:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x31d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086570:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x31d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086583:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000de5caded/0x4605c7c97e1bbd16 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1e9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4b0106 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086596:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3d5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086612:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3d5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086633:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000617603af/0x4605c7c97e1bbcfa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1e8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4b00b2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086652:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3d3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086664:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3d3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086679:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000016128b0b/0x4605c7c97e1bbcde lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1e7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4b005e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086691:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3d1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086701:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3d1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086714:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000023a72971/0x4605c7c97e1bbcc2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1e6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4b000a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086727:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3cf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086736:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3cf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086749:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006e2ff865/0x4605c7c97e1bb2ce lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x18b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae22e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086761:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x319:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086771:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x319:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086785:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b86cafd2/0x4605c7c97e1bbca6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1e5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4affb6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086797:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3cd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086807:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3cd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086820:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bc1017f1/0x4605c7c97e1bad02 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x156:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad0ca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086832:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2af:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086858:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2af:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086873:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004a2a7c38/0x4605c7c97e1bbc8a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1e4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aff62 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086901:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3cb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086913:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3cb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086927:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000103f84a6/0x4605c7c97e1ba9d6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x139:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac746 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086940:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x275:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086950:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x275:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086963:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c4e221a1/0x4605c7c97e1bbc52 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1e2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4afeba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.086975:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3c7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.086985:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3c7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.086998:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000abdac6ac/0x4605c7c97e1bb296 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x189:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae186 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087010:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x315:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087021:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x315:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087040:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003758ddce/0x4605c7c97e1bace6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x155:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad076 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087061:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2ad:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087078:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2ad:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087096:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000de40d03d/0x4605c7c97e1bbc1a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1e0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4afe12 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087111:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3c3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087123:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3c3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087136:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004fe85604/0x4605c7c97e1bbbfe lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1df:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4afdbe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087149:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3c1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087160:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3c1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087173:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006ace85ef/0x4605c7c97e1bbbe2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1de:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4afd6a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087190:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3bf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087207:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3bf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087228:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058ea240b/0x4605c7c97e1bb25e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x187:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae0de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087242:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x311:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087255:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x311:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087268:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000550c8fb0/0x4605c7c97e1bbbaa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1dc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4afcc2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087281:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3bb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087292:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3bb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087305:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003fe4a50c/0x4605c7c97e1ba9ba lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x138:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac6f2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087317:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x273:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087327:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x273:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087340:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006dec850f/0x4605c7c97e1bbb8e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1db:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4afc6e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087352:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3b9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087362:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3b9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087375:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001508a42d/0x4605c7c97e1bb242 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x186:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae08a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087388:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x30f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087397:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x30f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087410:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002a9b5648/0x4605c7c97e1bbb72 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1da:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4afc1a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087422:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3b7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087434:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3b7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087447:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005199bbbf/0x4605c7c97e1bacae lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x153:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4acfce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087459:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2a9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087470:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2a9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087484:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c0587592/0x4605c7c97e1bbb56 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1d9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4afbc6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087505:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x3b5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087522:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x3b5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087540:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000cce7ece/0x4605c7c97e1bb226 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x185:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae036 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087554:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x30d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087565:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x30d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087578:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005908f60f/0x4605c7c97e1ba6e2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x11e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4abe6a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087590:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x23f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087603:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x23f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087624:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b048fe02/0x4605c7c97e1bb20a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x184:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4adfe2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087645:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x30b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087658:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x30b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087672:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cfd044cd/0x4605c7c97e1bac92 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x152:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4acf7a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087686:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2a7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087697:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2a7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087710:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e0042d45/0x4605c7c97e1bb1ee lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x183:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4adf8e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087722:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x309:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087733:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x309:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087746:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ef821efd/0x4605c7c97e1ba99e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x137:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac69e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087758:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x271:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087767:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x271:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087780:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c1b8ec95/0x4605c7c97e1bb1d2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x182:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4adf3a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087792:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x307:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087802:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x307:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087814:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fda70a97/0x4605c7c97e1bac76 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x151:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4acf26 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087826:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2a5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087846:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2a5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087862:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000044eebd9c/0x4605c7c97e1bb1b6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x181:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4adee6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087888:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x305:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087901:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x305:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087915:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c420139b/0x4605c7c97e1ba7c2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x126:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac10a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087929:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x24f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087939:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x24f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087952:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000024e647e6/0x4605c7c97e1bb19a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x180:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ade92 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087964:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x303:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.087973:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x303:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.087986:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006fc3d2f1/0x4605c7c97e1bba22 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af82a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.087998:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x39f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088009:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x39f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088022:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000024111519/0x4605c7c97e1bac5a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x150:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aced2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088043:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2a3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088060:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2a3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088079:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cc3e6367/0x4605c7c97e1bba06 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af7d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088093:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x39d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088106:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x39d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088119:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000002a6b14/0x4605c7c97e1bb17e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x17f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ade3e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088132:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x301:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088142:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x301:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088154:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b3836a89/0x4605c7c97e1bb9ea lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af782 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088166:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x39b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088176:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x39b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088194:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000026f6f927/0x4605c7c97e1ba982 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x136:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac64a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088216:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x26f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088234:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x26f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088255:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000af62b26c/0x4605c7c97e1bb9ce lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af72e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088277:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x399:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088292:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x399:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088306:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001729ae97/0x4605c7c97e1bb162 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x17e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4addea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088320:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2ff:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088330:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2ff:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088343:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000239d571f/0x4605c7c97e1bb996 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1c9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af686 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088356:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x395:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088369:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x395:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088391:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005b109136/0x4605c7c97e1bb146 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x17d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4add96 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088413:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2fd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088430:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2fd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088453:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000066e244c8/0x4605c7c97e1bb97a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1c8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af632 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088471:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x393:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088487:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x393:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088509:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000faa9a2b1/0x4605c7c97e1ba5ca lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x114:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4abb22 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088529:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x22b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088542:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x22b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088557:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003b8e5f79/0x4605c7c97e1bb95e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1c7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af5de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088570:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x391:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088580:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x391:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088594:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002e029937/0x4605c7c97e1bb12a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x17c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4add42 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088614:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2fb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088630:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2fb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088648:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008fff66de/0x4605c7c97e1bb942 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1c6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af58a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088663:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x38f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088676:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x38f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088689:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000950ab880/0x4605c7c97e1bac22 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x14e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ace2a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088701:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x29f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088711:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x29f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088724:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000fa8322f/0x4605c7c97e1bb926 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1c5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af536 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088737:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x38d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088747:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x38d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088760:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003ec13691/0x4605c7c97e1bb90a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1c4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af4e2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088772:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x38b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088784:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x38b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088807:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c9ee59cd/0x4605c7c97e1bb8ee lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1c3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af48e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088826:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x389:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088863:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x389:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088902:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000812869c2/0x4605c7c97e1bb8d2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af43a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088923:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x387:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088938:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x387:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.088958:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bca7647c/0x4605c7c97e1bac06 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x14d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4acdd6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.088979:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x29d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.088995:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x29d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089015:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000002b9085/0x4605c7c97e1bb8b6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1c1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af3e6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089036:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x385:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089056:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x385:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089076:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b20b7c1f/0x4605c7c97e1bb0d6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x179:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4adc46 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089098:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2f5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089114:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2f5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089132:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009657afe5/0x4605c7c97e1bb87e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1bf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af33e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089146:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x381:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089158:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x381:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089171:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000112482fa/0x4605c7c97e1bb0ba lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x178:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4adbf2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089188:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2f3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089204:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2f3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089225:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000000c3c5ff/0x4605c7c97e1babea lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x14c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4acd82 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089242:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x29b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089254:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x29b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089268:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f2dee099/0x4605c7c97e1bb846 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1bd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af296 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089281:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x37d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089290:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x37d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089303:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e73ab13/0x4605c7c97e1bb09e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x177:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4adb9e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089316:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2f1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089325:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2f1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089338:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091ed0275/0x4605c7c97e1ba94a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x134:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac5a2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089350:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x26b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089361:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x26b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089381:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d55e254c/0x4605c7c97e1bb80e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1bb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af1ee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089401:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x379:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089413:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x379:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089428:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000005081b4a/0x4605c7c97e1bb082 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x176:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4adb4a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089443:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2ef:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089458:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2ef:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089479:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007c73c2ab/0x4605c7c97e1babce lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x14b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4acd2e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089501:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x299:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089517:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x299:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089537:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7e0281b/0x4605c7c97e1bb7d6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1b9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af146 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089552:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x375:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089564:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x375:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089577:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d26602ab/0x4605c7c97e1bb066 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x175:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4adaf6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089590:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2ed:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089602:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2ed:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089623:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011062e6d/0x4605c7c97e1ba6c6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x11d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4abe16 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089645:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x23d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089658:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x23d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089672:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000001a43a16/0x4605c7c97e1bb79e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1b7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af09e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089685:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x371:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089696:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x371:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089709:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005c3c7b72/0x4605c7c97e1bb782 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1b6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4af04a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089722:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x36f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089732:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x36f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089745:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008e47f008/0x4605c7c97e1bb766 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1b5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aeff6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089757:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x36d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089766:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x36d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089780:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003c3df6c1/0x4605c7c97e1bb02e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x173:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ada4e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089801:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2e9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089816:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2e9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089834:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d78702d0/0x4605c7c97e1bb74a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1b4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aefa2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089862:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x36b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089888:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x36b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089903:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000014b58190/0x4605c7c97e1ba92e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x133:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac54e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089917:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x269:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089929:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x269:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089941:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a21d60e7/0x4605c7c97e1bb712 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1b2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aeefa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089954:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x367:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089964:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x367:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.089976:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000adeec58e/0x4605c7c97e1bab96 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x149:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4acc86 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.089989:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x295:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.089999:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x295:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090012:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000079f9bf6b/0x4605c7c97e1bb6f6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1b1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aeea6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090025:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x365:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090040:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x365:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090063:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000629b598a/0x4605c7c97e1baff6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x171:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad9a6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090085:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2e5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090102:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2e5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090123:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000caed67bb/0x4605c7c97e1bb6da lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1b0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aee52 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090138:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x363:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090149:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x363:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090163:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bcb29ae6/0x4605c7c97e1bb6be lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1af:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aedfe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090175:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x361:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090190:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x361:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090212:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000930555ec/0x4605c7c97e1bb6a2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1ae:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aedaa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090234:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x35f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090251:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x35f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090267:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000868327c9/0x4605c7c97e1bb686 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1ad:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aed56 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090281:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x35d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090292:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x35d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090306:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002b52a7c0/0x4605c7c97e1bb66a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1ac:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aed02 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090318:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x35b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090329:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x35b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090342:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f81291be/0x4605c7c97e1bb64e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1ab:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aecae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090355:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x359:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090364:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x359:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090377:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000038b47fd8/0x4605c7c97e1bb632 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1aa:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aec5a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090389:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x357:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090399:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x357:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090412:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000077ac7f47/0x4605c7c97e1bb616 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1a9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aec06 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090424:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x355:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090433:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x355:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090446:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003b08e68c/0x4605c7c97e1bb5fa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1a8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aebb2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090458:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x353:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090468:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x353:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090480:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2e0f464/0x4605c7c97e1bb5de lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1a7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aeb5e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090493:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x351:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090503:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x351:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090516:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000eed9ef14/0x4605c7c97e1bb5c2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1a6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aeb0a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090529:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x34f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090538:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x34f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090550:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7bf1831/0x4605c7c97e1bb5a6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1a5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aeab6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090562:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x34d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090573:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x34d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090587:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000057997c2e/0x4605c7c97e1bb58a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1a4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aea62 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090603:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x34b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090619:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x34b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090640:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006a8b96dd/0x4605c7c97e1bb56e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1a3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aea0e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090657:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x349:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090674:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x349:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090696:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000016ff7546/0x4605c7c97e1bb552 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1a2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae9ba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090716:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x347:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090733:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x347:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090755:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000013a04ab/0x4605c7c97e1bb536 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1a1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae966 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090770:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x345:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090784:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x345:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090805:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000053feb2e1/0x4605c7c97e1bb51a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x1a0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae912 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090827:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x343:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090851:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x343:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090869:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7fa819a/0x4605c7c97e1bb4fe lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x19f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae8be expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090899:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x341:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090911:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x341:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090925:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000024d6ae0c/0x4605c7c97e1bab0a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x144:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4acae2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090938:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x28b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090949:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x28b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.090965:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000defa51e3/0x4605c7c97e1baec2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x166:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad60a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.090977:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2cf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.090987:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2cf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091000:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000956a613b/0x4605c7c97e1bae8a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x164:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad562 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091012:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2cb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091022:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2cb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091042:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000089b2da7a/0x4605c7c97e1baa9a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x140:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac992 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091063:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x283:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091082:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x283:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091103:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000017ada4a3/0x4605c7c97e1baa7e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x13f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac93e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091123:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x281:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091137:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x281:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091157:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008a85cb9c/0x4605c7c97e1baa62 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x13e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac8ea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091175:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x27f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091191:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x27f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091212:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d1233094/0x4605c7c97e1badfe lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x15f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad3be expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091231:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2c1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091246:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2c1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091268:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000393e86fb/0x4605c7c97e1bb4aa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x19c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae7c2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091290:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x33b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091309:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x33b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091339:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000354715db/0x4605c7c97e1baa46 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x13d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac896 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091354:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x27d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091366:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x27d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091380:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000094b0e1c1/0x4605c7c97e1badc6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x15d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad316 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091392:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2bd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091402:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2bd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091415:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e9ea91ce/0x4605c7c97e1ba86a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x12c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac302 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091427:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x25b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091437:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x25b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091450:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000068e6d36a/0x4605c7c97e1badaa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x15c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad2c2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091462:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2bb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091471:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2bb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091484:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d6d88ce5/0x4605c7c97e1baa2a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x13c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac842 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091496:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x27b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091506:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x27b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091519:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bde6697d/0x4605c7c97e1bad8e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x15b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad26e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091532:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2b9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091541:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2b9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091555:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000efd01301/0x4605c7c97e1ba68e lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x11b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4abd6e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091567:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x239:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091577:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x239:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091590:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002ce081f1/0x4605c7c97e1bad56 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x159:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad1c6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091608:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2b5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091625:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2b5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091646:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e11d290/0x4605c7c97e1bb35a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x190:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ae3d2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091661:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x323:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091673:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x323:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091687:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003b838344/0x4605c7c97e1baede lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x167:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad65e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091699:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2d1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091709:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2d1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091721:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001f56ffcf/0x4605c7c97e1baaee lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x143:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aca8e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091734:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x289:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091743:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x289:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091756:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a527bd73/0x4605c7c97e1ba8be lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x12f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac3fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091768:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x261:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091779:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x261:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091799:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f066339b/0x4605c7c97e1baab6 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x141:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac9e6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091822:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x285:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091834:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x285:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091862:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e0bca7d7/0x4605c7c97e1ba752 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x122:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4abfba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091890:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x247:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091903:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x247:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091918:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000030e06fa3/0x4605c7c97e1ba8a2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x12e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ac3aa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091932:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x25f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091941:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x25f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091954:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006b444f29/0x4605c7c97e1bae52 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x162:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad4ba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.091967:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2c7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.091977:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2c7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.091989:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ebe9b562/0x4605c7c97e1bae36 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x161:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad466 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.092002:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2c5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.092012:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2c5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.092026:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000022e395f3/0x4605c7c97e1bae1a lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x160:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4ad412 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.092047:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x2c3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.092063:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x2c3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.092081:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000029f229e5/0x4605c7c97e1ba6aa lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x11c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4abdc2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.092095:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x23b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.092107:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x23b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.092119:0:5598:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d91a432d/0x4605c7c97e1baad2 lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x142:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400000000 nid: local remote: 0x9383f985bc4aca3a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201702.092132:0:5598:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x287:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788201702.092141:0:5598:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x287:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1788201702.092176:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bed84690/0x4605c7c97e1bade2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x15e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad36a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092181:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000276613e8/0x4605c7c97e1bb82a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1bc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af242 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092189:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004e17030d/0x4605c7c97e1baa0e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x13b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac7ee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092194:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cbad7b7/0x4605c7c97e1bbb3a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1d8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afb72 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092198:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000ed51db4/0x4605c7c97e1bac3e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x14f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ace7e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092202:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005bceb709/0x4605c7c97e1ba7a6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x125:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac0b6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092206:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005aec1927/0x4605c7c97e1ba886 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x12d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac356 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092210:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fc124b78/0x4605c7c97e1bad72 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x15a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad21a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092214:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000022a4dde9/0x4605c7c97e1bb33e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x18f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae37e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092218:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bfa2f08d/0x4605c7c97e1bbd32 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1ea:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b015a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092222:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000013c8f5e9/0x4605c7c97e1bb2ea lrc: 1/0,0 mode: --/PW res: [0x240000400:0x18c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae282 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092230:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005246a2b4/0x4605c7c97e1bb2b2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x18a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae1da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092234:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000024f5da4d/0x4605c7c97e1bbc36 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1e1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afe66 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092238:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005661ba59/0x4605c7c97e1bb27a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x188:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae132 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092242:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b5f9e4b1/0x4605c7c97e1bbbc6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1dd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afd16 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092247:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009e0a5d36/0x4605c7c97e1bb9b2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af6da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092251:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000368e7f31/0x4605c7c97e1bb0f2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x17a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adc9a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092255:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c99af6d6/0x4605c7c97e1bb7ba lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1b8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af0f2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092259:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007e84d9bc/0x4605c7c97e1babb2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x14a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4accda expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092263:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001c7d459f/0x4605c7c97e1bab5e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x147:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acbde expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092271:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007814fa75/0x4605c7c97e1ba656 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x119:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abcc6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092275:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e6acb7a1/0x4605c7c97e1ba76e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x123:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac00e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092279:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fd4f04bc/0x4605c7c97e1ba8da lrc: 1/0,0 mode: --/PW res: [0x240000400:0x130:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac452 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092283:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000de1f3c5/0x4605c7c97e1bae6e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x163:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad50e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092287:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c5544a0c/0x4605c7c97e1ba736 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x121:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abf66 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092290:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ac29708b/0x4605c7c97e1bb41e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x197:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae61e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092294:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b6fbaa07/0x4605c7c97e1bb392 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x192:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae47a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092298:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000104ead1a/0x4605c7c97e1ba84e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x12b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac2ae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092302:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000efb3dfe1/0x4605c7c97e1bb322 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x18e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae32a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092310:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001c5bb60f/0x4605c7c97e1bad1e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x157:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad11e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092314:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000d743726/0x4605c7c97e1ba71a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x120:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abf12 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092318:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e97d125/0x4605c7c97e1bbc6e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1e3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aff0e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092322:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000032bcdd95/0x4605c7c97e1ba7de lrc: 1/0,0 mode: --/PW res: [0x240000400:0x127:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac15e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092326:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000040396122/0x4605c7c97e1bacca lrc: 1/0,0 mode: --/PW res: [0x240000400:0x154:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad022 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092330:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000854cfd71/0x4605c7c97e1bbb1e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1d7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afb1e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092334:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005340f48b/0x4605c7c97e1bbb02 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1d6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afaca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092338:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000062656217/0x4605c7c97e1bbae6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1d5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afa76 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092342:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f1570ec1/0x4605c7c97e1bbaca lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1d4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afa22 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092349:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006a7b1128/0x4605c7c97e1bbaae lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1d3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af9ce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092354:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000235e5f70/0x4605c7c97e1bba92 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1d2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af97a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092358:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000066b8ebd6/0x4605c7c97e1bba76 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1d1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af926 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092362:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007e635a94/0x4605c7c97e1bba5a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af8d2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092366:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009055f3c2/0x4605c7c97e1bba3e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1cf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af87e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092370:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000567b46ee/0x4605c7c97e1bb10e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x17b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adcee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092374:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005b2da7a3/0x4605c7c97e1ba966 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x135:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac5f6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092378:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e46c2198/0x4605c7c97e1bb89a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1c0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af392 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092382:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b0abece8/0x4605c7c97e1bb862 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1be:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af2ea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092389:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f59864fa/0x4605c7c97e1bb7f2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1ba:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af19a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092393:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000475942c0/0x4605c7c97e1bb04a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x174:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adaa2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092397:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f2835f0f/0x4605c7c97e1bb72e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1b3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aef4e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092401:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c8673266/0x4605c7c97e1bb012 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x172:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad9fa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092405:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000827b78a6/0x4605c7c97e1ba78a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x124:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac062 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092409:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009819f599/0x4605c7c97e1bafda lrc: 1/0,0 mode: --/PW res: [0x240000400:0x170:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad952 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092413:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002ce3e984/0x4605c7c97e1bab7a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x148:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acc32 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092417:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000aa6e73d1/0x4605c7c97e1bafbe lrc: 1/0,0 mode: --/PW res: [0x240000400:0x16f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad8fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092421:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000023f07ef9/0x4605c7c97e1ba912 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x132:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac4fa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092428:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003fd95c5c/0x4605c7c97e1bafa2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x16e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad8aa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092433:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bf9dcc7a/0x4605c7c97e1baf86 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x16d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad856 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092436:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007408a795/0x4605c7c97e1baf6a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x16c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad802 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092440:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009f8d40b4/0x4605c7c97e1bab42 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x146:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acb8a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092444:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004197033e/0x4605c7c97e1baf4e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x16b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad7ae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092448:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ae1be017/0x4605c7c97e1ba8f6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x131:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac4a6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092452:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000367f034/0x4605c7c97e1baf32 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x16a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad75a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092456:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7408653/0x4605c7c97e1bab26 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x145:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acb36 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092460:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000137bbb8d/0x4605c7c97e1baf16 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x169:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad706 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092467:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000025757b46/0x4605c7c97e1baefa lrc: 1/0,0 mode: --/PW res: [0x240000400:0x168:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad6b2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092471:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ee55191d/0x4605c7c97e1baea6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x165:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad5b6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092475:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000098620651/0x4605c7c97e1bb4e2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x19e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae86a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092479:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006e6fa8b/0x4605c7c97e1bb4c6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x19d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae816 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092483:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006f703151/0x4605c7c97e1bbe66 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1f5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b04f6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092487:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000100df4ce/0x4605c7c97e1bb48e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x19b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae76e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092491:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000842865e9/0x4605c7c97e1bbe4a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1f4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b04a2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092495:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000074b50444/0x4605c7c97e1bbe2e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1f3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b044e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092499:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006e3de7cc/0x4605c7c97e1bb472 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x19a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae71a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092506:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000021db2522/0x4605c7c97e1bb456 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x199:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae6c6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092510:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000048a5c5a4/0x4605c7c97e1bb43a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x198:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae672 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092514:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005ecb8ec4/0x4605c7c97e1bb402 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x196:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae5ca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092518:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000671e6492/0x4605c7c97e1bb3e6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x195:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae576 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092522:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e859d16e/0x4605c7c97e1bb3ca lrc: 1/0,0 mode: --/PW res: [0x240000400:0x194:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae522 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092526:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a3abdab2/0x4605c7c97e1bb3ae lrc: 1/0,0 mode: --/PW res: [0x240000400:0x193:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae4ce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092530:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000023f36339/0x4605c7c97e1bbe12 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1f2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b03fa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092534:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000134a3795/0x4605c7c97e1bbdf6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1f1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b03a6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092538:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000a600c03/0x4605c7c97e1bbdda lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1f0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b0352 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092545:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002e8b241c/0x4605c7c97e1bbdbe lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1ef:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b02fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092549:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000013d10d82/0x4605c7c97e1bb376 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x191:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae426 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092554:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000062a5f2b4/0x4605c7c97e1bbda2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1ee:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b02aa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092558:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ac503f81/0x4605c7c97e1bbd86 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1ed:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b0256 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092561:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000063f19c32/0x4605c7c97e1bad3a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x158:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad172 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092565:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000be7eeb64/0x4605c7c97e1bbd6a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1ec:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b0202 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092570:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ff767e23/0x4605c7c97e1bbd4e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1eb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b01ae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092574:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c26ae6a4/0x4605c7c97e1ba9f2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x13a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac79a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092578:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003db3ea37/0x4605c7c97e1bb306 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x18d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae2d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092585:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000de5caded/0x4605c7c97e1bbd16 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1e9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b0106 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092589:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000617603af/0x4605c7c97e1bbcfa lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1e8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b00b2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092595:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000016128b0b/0x4605c7c97e1bbcde lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1e7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b005e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092601:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000023a72971/0x4605c7c97e1bbcc2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1e6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b000a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092608:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006e2ff865/0x4605c7c97e1bb2ce lrc: 1/0,0 mode: --/PW res: [0x240000400:0x18b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae22e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092615:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b86cafd2/0x4605c7c97e1bbca6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1e5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4affb6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092622:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bc1017f1/0x4605c7c97e1bad02 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x156:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad0ca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092630:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004a2a7c38/0x4605c7c97e1bbc8a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1e4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aff62 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092638:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000103f84a6/0x4605c7c97e1ba9d6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x139:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac746 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092646:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c4e221a1/0x4605c7c97e1bbc52 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1e2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afeba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092650:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000abdac6ac/0x4605c7c97e1bb296 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x189:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae186 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092654:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003758ddce/0x4605c7c97e1bace6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x155:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad076 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092659:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000de40d03d/0x4605c7c97e1bbc1a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1e0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afe12 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092663:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004fe85604/0x4605c7c97e1bbbfe lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1df:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afdbe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092667:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006ace85ef/0x4605c7c97e1bbbe2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1de:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afd6a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092671:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058ea240b/0x4605c7c97e1bb25e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x187:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae0de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092675:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000550c8fb0/0x4605c7c97e1bbbaa lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1dc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afcc2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092679:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003fe4a50c/0x4605c7c97e1ba9ba lrc: 1/0,0 mode: --/PW res: [0x240000400:0x138:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac6f2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092686:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006dec850f/0x4605c7c97e1bbb8e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1db:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afc6e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092690:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001508a42d/0x4605c7c97e1bb242 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x186:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae08a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092694:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002a9b5648/0x4605c7c97e1bbb72 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1da:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afc1a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092698:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005199bbbf/0x4605c7c97e1bacae lrc: 1/0,0 mode: --/PW res: [0x240000400:0x153:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acfce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092702:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c0587592/0x4605c7c97e1bbb56 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1d9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afbc6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092706:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000cce7ece/0x4605c7c97e1bb226 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x185:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae036 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092710:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005908f60f/0x4605c7c97e1ba6e2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x11e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abe6a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092714:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b048fe02/0x4605c7c97e1bb20a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x184:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adfe2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092717:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cfd044cd/0x4605c7c97e1bac92 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x152:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acf7a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092724:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e0042d45/0x4605c7c97e1bb1ee lrc: 1/0,0 mode: --/PW res: [0x240000400:0x183:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adf8e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092729:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ef821efd/0x4605c7c97e1ba99e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x137:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac69e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092733:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c1b8ec95/0x4605c7c97e1bb1d2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x182:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adf3a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092737:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fda70a97/0x4605c7c97e1bac76 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x151:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acf26 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092741:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000044eebd9c/0x4605c7c97e1bb1b6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x181:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adee6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092746:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c420139b/0x4605c7c97e1ba7c2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x126:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac10a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092753:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000024e647e6/0x4605c7c97e1bb19a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x180:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ade92 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092759:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006fc3d2f1/0x4605c7c97e1bba22 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af82a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092763:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000024111519/0x4605c7c97e1bac5a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x150:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aced2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092773:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cc3e6367/0x4605c7c97e1bba06 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af7d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092780:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000002a6b14/0x4605c7c97e1bb17e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x17f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ade3e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092786:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b3836a89/0x4605c7c97e1bb9ea lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af782 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092792:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000026f6f927/0x4605c7c97e1ba982 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x136:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac64a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092798:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000af62b26c/0x4605c7c97e1bb9ce lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af72e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092804:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001729ae97/0x4605c7c97e1bb162 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x17e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4addea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092811:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000239d571f/0x4605c7c97e1bb996 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1c9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af686 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092817:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005b109136/0x4605c7c97e1bb146 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x17d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4add96 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092824:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000066e244c8/0x4605c7c97e1bb97a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1c8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af632 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092833:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000faa9a2b1/0x4605c7c97e1ba5ca lrc: 1/0,0 mode: --/PW res: [0x240000400:0x114:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abb22 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092851:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003b8e5f79/0x4605c7c97e1bb95e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1c7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af5de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092857:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002e029937/0x4605c7c97e1bb12a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x17c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4add42 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092863:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008fff66de/0x4605c7c97e1bb942 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1c6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af58a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092869:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000950ab880/0x4605c7c97e1bac22 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x14e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ace2a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092874:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000fa8322f/0x4605c7c97e1bb926 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1c5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af536 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092897:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003ec13691/0x4605c7c97e1bb90a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1c4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af4e2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092903:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c9ee59cd/0x4605c7c97e1bb8ee lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1c3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af48e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092909:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000812869c2/0x4605c7c97e1bb8d2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af43a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092919:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bca7647c/0x4605c7c97e1bac06 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x14d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acdd6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092926:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000002b9085/0x4605c7c97e1bb8b6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1c1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af3e6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092932:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b20b7c1f/0x4605c7c97e1bb0d6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x179:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adc46 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092940:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009657afe5/0x4605c7c97e1bb87e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1bf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af33e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092947:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000112482fa/0x4605c7c97e1bb0ba lrc: 1/0,0 mode: --/PW res: [0x240000400:0x178:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adbf2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092952:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000000c3c5ff/0x4605c7c97e1babea lrc: 1/0,0 mode: --/PW res: [0x240000400:0x14c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acd82 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092957:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f2dee099/0x4605c7c97e1bb846 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1bd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af296 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092961:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e73ab13/0x4605c7c97e1bb09e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x177:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adb9e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092965:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091ed0275/0x4605c7c97e1ba94a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x134:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac5a2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092972:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d55e254c/0x4605c7c97e1bb80e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1bb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af1ee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092976:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000005081b4a/0x4605c7c97e1bb082 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x176:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adb4a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092980:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007c73c2ab/0x4605c7c97e1babce lrc: 1/0,0 mode: --/PW res: [0x240000400:0x14b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acd2e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092984:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7e0281b/0x4605c7c97e1bb7d6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1b9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af146 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092988:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d26602ab/0x4605c7c97e1bb066 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x175:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adaf6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092991:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011062e6d/0x4605c7c97e1ba6c6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x11d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abe16 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092995:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000001a43a16/0x4605c7c97e1bb79e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1b7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af09e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.092999:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005c3c7b72/0x4605c7c97e1bb782 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1b6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af04a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093003:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008e47f008/0x4605c7c97e1bb766 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1b5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aeff6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093010:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003c3df6c1/0x4605c7c97e1bb02e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x173:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ada4e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093014:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d78702d0/0x4605c7c97e1bb74a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1b4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aefa2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093018:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000014b58190/0x4605c7c97e1ba92e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x133:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac54e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093022:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a21d60e7/0x4605c7c97e1bb712 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1b2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aeefa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093029:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000adeec58e/0x4605c7c97e1bab96 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x149:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acc86 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093035:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000079f9bf6b/0x4605c7c97e1bb6f6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1b1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aeea6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093041:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000629b598a/0x4605c7c97e1baff6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x171:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad9a6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093048:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000caed67bb/0x4605c7c97e1bb6da lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1b0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aee52 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093055:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bcb29ae6/0x4605c7c97e1bb6be lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1af:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aedfe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093066:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000930555ec/0x4605c7c97e1bb6a2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1ae:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aedaa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093073:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000868327c9/0x4605c7c97e1bb686 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1ad:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aed56 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093079:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002b52a7c0/0x4605c7c97e1bb66a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1ac:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aed02 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093086:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f81291be/0x4605c7c97e1bb64e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1ab:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aecae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093094:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000038b47fd8/0x4605c7c97e1bb632 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1aa:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aec5a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093099:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000077ac7f47/0x4605c7c97e1bb616 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1a9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aec06 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093106:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003b08e68c/0x4605c7c97e1bb5fa lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1a8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aebb2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093113:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2e0f464/0x4605c7c97e1bb5de lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1a7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aeb5e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093120:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000eed9ef14/0x4605c7c97e1bb5c2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1a6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aeb0a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093131:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7bf1831/0x4605c7c97e1bb5a6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1a5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aeab6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093139:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000057997c2e/0x4605c7c97e1bb58a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1a4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aea62 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093146:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006a8b96dd/0x4605c7c97e1bb56e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1a3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aea0e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093150:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000016ff7546/0x4605c7c97e1bb552 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1a2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae9ba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093154:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000013a04ab/0x4605c7c97e1bb536 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1a1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae966 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093158:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000053feb2e1/0x4605c7c97e1bb51a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x1a0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae912 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093162:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7fa819a/0x4605c7c97e1bb4fe lrc: 1/0,0 mode: --/PW res: [0x240000400:0x19f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae8be expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093166:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000024d6ae0c/0x4605c7c97e1bab0a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x144:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acae2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093172:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000defa51e3/0x4605c7c97e1baec2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x166:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad60a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093183:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000956a613b/0x4605c7c97e1bae8a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x164:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad562 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093189:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000089b2da7a/0x4605c7c97e1baa9a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x140:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac992 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093195:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000017ada4a3/0x4605c7c97e1baa7e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x13f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac93e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093202:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008a85cb9c/0x4605c7c97e1baa62 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x13e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac8ea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093209:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d1233094/0x4605c7c97e1badfe lrc: 1/0,0 mode: --/PW res: [0x240000400:0x15f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad3be expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093216:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000393e86fb/0x4605c7c97e1bb4aa lrc: 1/0,0 mode: --/PW res: [0x240000400:0x19c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae7c2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093222:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000354715db/0x4605c7c97e1baa46 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x13d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac896 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093227:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000094b0e1c1/0x4605c7c97e1badc6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x15d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad316 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093233:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e9ea91ce/0x4605c7c97e1ba86a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x12c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac302 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093244:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000068e6d36a/0x4605c7c97e1badaa lrc: 1/0,0 mode: --/PW res: [0x240000400:0x15c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad2c2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093250:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d6d88ce5/0x4605c7c97e1baa2a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x13c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac842 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093256:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bde6697d/0x4605c7c97e1bad8e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x15b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad26e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093263:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000efd01301/0x4605c7c97e1ba68e lrc: 1/0,0 mode: --/PW res: [0x240000400:0x11b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abd6e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093270:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002ce081f1/0x4605c7c97e1bad56 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x159:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad1c6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093277:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e11d290/0x4605c7c97e1bb35a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x190:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae3d2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093284:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003b838344/0x4605c7c97e1baede lrc: 1/0,0 mode: --/PW res: [0x240000400:0x167:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad65e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093292:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001f56ffcf/0x4605c7c97e1baaee lrc: 1/0,0 mode: --/PW res: [0x240000400:0x143:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aca8e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093296:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a527bd73/0x4605c7c97e1ba8be lrc: 1/0,0 mode: --/PW res: [0x240000400:0x12f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac3fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093303:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f066339b/0x4605c7c97e1baab6 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x141:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac9e6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093307:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e0bca7d7/0x4605c7c97e1ba752 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x122:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abfba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093311:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000030e06fa3/0x4605c7c97e1ba8a2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x12e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac3aa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093315:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006b444f29/0x4605c7c97e1bae52 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x162:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad4ba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093319:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ebe9b562/0x4605c7c97e1bae36 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x161:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad466 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093322:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000022e395f3/0x4605c7c97e1bae1a lrc: 1/0,0 mode: --/PW res: [0x240000400:0x160:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad412 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093326:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000029f229e5/0x4605c7c97e1ba6aa lrc: 1/0,0 mode: --/PW res: [0x240000400:0x11c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abdc2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093329:0:5598:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d91a432d/0x4605c7c97e1baad2 lrc: 1/0,0 mode: --/PW res: [0x240000400:0x142:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aca3a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093333:0:5598:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 217 locks packed 00010000:00010000:1.0:1788201702.093358:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bed84690/0x4605c7c97e1bade2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x15e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad36a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093374:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000276613e8/0x4605c7c97e1bb82a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1bc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af242 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093383:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004e17030d/0x4605c7c97e1baa0e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x13b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac7ee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093392:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004cbad7b7/0x4605c7c97e1bbb3a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1d8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afb72 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093401:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000ed51db4/0x4605c7c97e1bac3e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x14f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ace7e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093411:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005bceb709/0x4605c7c97e1ba7a6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x125:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac0b6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000100:00100000:0.0:1788201702.093418:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442801024:192.168.201.130@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788201702.093425:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005aec1927/0x4605c7c97e1ba886 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x12d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac356 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093438:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fc124b78/0x4605c7c97e1bad72 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x15a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad21a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093449:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000022a4dde9/0x4605c7c97e1bb33e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x18f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae37e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093466:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bfa2f08d/0x4605c7c97e1bbd32 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1ea:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b015a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093478:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000013c8f5e9/0x4605c7c97e1bb2ea lrc: 0/0,0 mode: --/PW res: [0x240000400:0x18c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae282 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093486:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005246a2b4/0x4605c7c97e1bb2b2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x18a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae1da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093494:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000024f5da4d/0x4605c7c97e1bbc36 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1e1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afe66 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093502:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005661ba59/0x4605c7c97e1bb27a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x188:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae132 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093509:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b5f9e4b1/0x4605c7c97e1bbbc6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1dd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afd16 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093518:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009e0a5d36/0x4605c7c97e1bb9b2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1ca:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af6da expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093527:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000368e7f31/0x4605c7c97e1bb0f2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x17a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adc9a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093539:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c99af6d6/0x4605c7c97e1bb7ba lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1b8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af0f2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093550:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007e84d9bc/0x4605c7c97e1babb2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x14a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4accda expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093562:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001c7d459f/0x4605c7c97e1bab5e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x147:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acbde expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093573:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007814fa75/0x4605c7c97e1ba656 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x119:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abcc6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093583:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e6acb7a1/0x4605c7c97e1ba76e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x123:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac00e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093590:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fd4f04bc/0x4605c7c97e1ba8da lrc: 0/0,0 mode: --/PW res: [0x240000400:0x130:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac452 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093600:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000de1f3c5/0x4605c7c97e1bae6e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x163:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad50e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093610:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c5544a0c/0x4605c7c97e1ba736 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x121:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abf66 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093626:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ac29708b/0x4605c7c97e1bb41e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x197:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae61e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093637:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b6fbaa07/0x4605c7c97e1bb392 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x192:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae47a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093645:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000104ead1a/0x4605c7c97e1ba84e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x12b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac2ae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093652:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000efb3dfe1/0x4605c7c97e1bb322 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x18e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae32a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093657:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001c5bb60f/0x4605c7c97e1bad1e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x157:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad11e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093664:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000d743726/0x4605c7c97e1ba71a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x120:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abf12 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093670:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e97d125/0x4605c7c97e1bbc6e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1e3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aff0e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093675:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000032bcdd95/0x4605c7c97e1ba7de lrc: 0/0,0 mode: --/PW res: [0x240000400:0x127:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac15e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093685:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000040396122/0x4605c7c97e1bacca lrc: 0/0,0 mode: --/PW res: [0x240000400:0x154:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad022 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093692:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000854cfd71/0x4605c7c97e1bbb1e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1d7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afb1e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093698:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005340f48b/0x4605c7c97e1bbb02 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1d6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afaca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093704:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000062656217/0x4605c7c97e1bbae6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1d5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afa76 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093710:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f1570ec1/0x4605c7c97e1bbaca lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1d4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afa22 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093717:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006a7b1128/0x4605c7c97e1bbaae lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1d3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af9ce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093723:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000235e5f70/0x4605c7c97e1bba92 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1d2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af97a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093729:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000066b8ebd6/0x4605c7c97e1bba76 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1d1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af926 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093739:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007e635a94/0x4605c7c97e1bba5a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1d0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af8d2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093745:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009055f3c2/0x4605c7c97e1bba3e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1cf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af87e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093750:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000567b46ee/0x4605c7c97e1bb10e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x17b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adcee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093758:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005b2da7a3/0x4605c7c97e1ba966 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x135:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac5f6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093764:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e46c2198/0x4605c7c97e1bb89a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1c0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af392 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093770:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b0abece8/0x4605c7c97e1bb862 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1be:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af2ea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093776:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f59864fa/0x4605c7c97e1bb7f2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1ba:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af19a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093785:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000475942c0/0x4605c7c97e1bb04a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x174:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adaa2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093801:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f2835f0f/0x4605c7c97e1bb72e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1b3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aef4e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093812:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c8673266/0x4605c7c97e1bb012 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x172:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad9fa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093822:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000827b78a6/0x4605c7c97e1ba78a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x124:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac062 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093829:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009819f599/0x4605c7c97e1bafda lrc: 0/0,0 mode: --/PW res: [0x240000400:0x170:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad952 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093847:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002ce3e984/0x4605c7c97e1bab7a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x148:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acc32 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093855:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000aa6e73d1/0x4605c7c97e1bafbe lrc: 0/0,0 mode: --/PW res: [0x240000400:0x16f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad8fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093862:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000023f07ef9/0x4605c7c97e1ba912 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x132:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac4fa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093869:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003fd95c5c/0x4605c7c97e1bafa2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x16e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad8aa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093904:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bf9dcc7a/0x4605c7c97e1baf86 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x16d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad856 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093911:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007408a795/0x4605c7c97e1baf6a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x16c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad802 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093918:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009f8d40b4/0x4605c7c97e1bab42 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x146:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acb8a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093927:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004197033e/0x4605c7c97e1baf4e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x16b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad7ae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093938:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ae1be017/0x4605c7c97e1ba8f6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x131:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac4a6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093946:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000367f034/0x4605c7c97e1baf32 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x16a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad75a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093951:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7408653/0x4605c7c97e1bab26 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x145:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acb36 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093958:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000137bbb8d/0x4605c7c97e1baf16 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x169:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad706 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093967:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000025757b46/0x4605c7c97e1baefa lrc: 0/0,0 mode: --/PW res: [0x240000400:0x168:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad6b2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093973:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ee55191d/0x4605c7c97e1baea6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x165:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad5b6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093979:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000098620651/0x4605c7c97e1bb4e2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x19e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae86a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093985:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000006e6fa8b/0x4605c7c97e1bb4c6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x19d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae816 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093991:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006f703151/0x4605c7c97e1bbe66 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1f5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b04f6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.093997:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000100df4ce/0x4605c7c97e1bb48e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x19b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae76e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094003:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000842865e9/0x4605c7c97e1bbe4a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1f4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b04a2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094010:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000074b50444/0x4605c7c97e1bbe2e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1f3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b044e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094020:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006e3de7cc/0x4605c7c97e1bb472 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x19a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae71a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094028:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000021db2522/0x4605c7c97e1bb456 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x199:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae6c6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094038:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000048a5c5a4/0x4605c7c97e1bb43a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x198:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae672 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094049:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005ecb8ec4/0x4605c7c97e1bb402 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x196:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae5ca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094059:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000671e6492/0x4605c7c97e1bb3e6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x195:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae576 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094069:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e859d16e/0x4605c7c97e1bb3ca lrc: 0/0,0 mode: --/PW res: [0x240000400:0x194:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae522 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094080:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a3abdab2/0x4605c7c97e1bb3ae lrc: 0/0,0 mode: --/PW res: [0x240000400:0x193:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae4ce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094089:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000023f36339/0x4605c7c97e1bbe12 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1f2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b03fa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094105:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000134a3795/0x4605c7c97e1bbdf6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1f1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b03a6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094115:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000a600c03/0x4605c7c97e1bbdda lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1f0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b0352 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094127:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002e8b241c/0x4605c7c97e1bbdbe lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1ef:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b02fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094133:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000013d10d82/0x4605c7c97e1bb376 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x191:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae426 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094139:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000062a5f2b4/0x4605c7c97e1bbda2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1ee:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b02aa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094145:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ac503f81/0x4605c7c97e1bbd86 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1ed:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b0256 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094152:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000063f19c32/0x4605c7c97e1bad3a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x158:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad172 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094158:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000be7eeb64/0x4605c7c97e1bbd6a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1ec:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b0202 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094168:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ff767e23/0x4605c7c97e1bbd4e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1eb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b01ae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094174:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c26ae6a4/0x4605c7c97e1ba9f2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x13a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac79a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094180:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003db3ea37/0x4605c7c97e1bb306 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x18d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae2d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094186:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000de5caded/0x4605c7c97e1bbd16 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1e9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b0106 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094192:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000617603af/0x4605c7c97e1bbcfa lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1e8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b00b2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094199:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000016128b0b/0x4605c7c97e1bbcde lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1e7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b005e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094205:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000023a72971/0x4605c7c97e1bbcc2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1e6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4b000a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094211:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006e2ff865/0x4605c7c97e1bb2ce lrc: 0/0,0 mode: --/PW res: [0x240000400:0x18b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae22e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094221:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b86cafd2/0x4605c7c97e1bbca6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1e5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4affb6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094227:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bc1017f1/0x4605c7c97e1bad02 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x156:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad0ca expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094233:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004a2a7c38/0x4605c7c97e1bbc8a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1e4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aff62 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094239:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000103f84a6/0x4605c7c97e1ba9d6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x139:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac746 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094244:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c4e221a1/0x4605c7c97e1bbc52 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1e2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afeba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094250:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000abdac6ac/0x4605c7c97e1bb296 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x189:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae186 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094256:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003758ddce/0x4605c7c97e1bace6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x155:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad076 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094263:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000de40d03d/0x4605c7c97e1bbc1a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1e0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afe12 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094272:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000004fe85604/0x4605c7c97e1bbbfe lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1df:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afdbe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094279:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006ace85ef/0x4605c7c97e1bbbe2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1de:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afd6a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094284:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000058ea240b/0x4605c7c97e1bb25e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x187:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae0de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094290:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000550c8fb0/0x4605c7c97e1bbbaa lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1dc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afcc2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094296:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003fe4a50c/0x4605c7c97e1ba9ba lrc: 0/0,0 mode: --/PW res: [0x240000400:0x138:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac6f2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094302:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006dec850f/0x4605c7c97e1bbb8e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1db:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afc6e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094308:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001508a42d/0x4605c7c97e1bb242 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x186:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae08a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094314:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002a9b5648/0x4605c7c97e1bbb72 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1da:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afc1a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094323:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005199bbbf/0x4605c7c97e1bacae lrc: 0/0,0 mode: --/PW res: [0x240000400:0x153:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acfce expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094330:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c0587592/0x4605c7c97e1bbb56 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1d9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4afbc6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094335:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000cce7ece/0x4605c7c97e1bb226 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x185:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae036 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094341:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005908f60f/0x4605c7c97e1ba6e2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x11e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abe6a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094347:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b048fe02/0x4605c7c97e1bb20a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x184:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adfe2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094352:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cfd044cd/0x4605c7c97e1bac92 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x152:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acf7a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094358:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e0042d45/0x4605c7c97e1bb1ee lrc: 0/0,0 mode: --/PW res: [0x240000400:0x183:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adf8e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094365:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ef821efd/0x4605c7c97e1ba99e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x137:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac69e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094375:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c1b8ec95/0x4605c7c97e1bb1d2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x182:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adf3a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094381:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000fda70a97/0x4605c7c97e1bac76 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x151:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acf26 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094387:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000044eebd9c/0x4605c7c97e1bb1b6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x181:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adee6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094394:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c420139b/0x4605c7c97e1ba7c2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x126:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac10a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094399:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000024e647e6/0x4605c7c97e1bb19a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x180:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ade92 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094406:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006fc3d2f1/0x4605c7c97e1bba22 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1ce:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af82a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094412:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000024111519/0x4605c7c97e1bac5a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x150:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aced2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094418:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000cc3e6367/0x4605c7c97e1bba06 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1cd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af7d6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094427:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000002a6b14/0x4605c7c97e1bb17e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x17f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ade3e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094432:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b3836a89/0x4605c7c97e1bb9ea lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1cc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af782 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094438:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000026f6f927/0x4605c7c97e1ba982 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x136:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac64a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094444:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000af62b26c/0x4605c7c97e1bb9ce lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1cb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af72e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094451:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001729ae97/0x4605c7c97e1bb162 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x17e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4addea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094457:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000239d571f/0x4605c7c97e1bb996 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1c9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af686 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094462:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005b109136/0x4605c7c97e1bb146 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x17d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4add96 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094469:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000066e244c8/0x4605c7c97e1bb97a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1c8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af632 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094478:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000faa9a2b1/0x4605c7c97e1ba5ca lrc: 0/0,0 mode: --/PW res: [0x240000400:0x114:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abb22 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094490:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003b8e5f79/0x4605c7c97e1bb95e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1c7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af5de expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094500:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002e029937/0x4605c7c97e1bb12a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x17c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4add42 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094511:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008fff66de/0x4605c7c97e1bb942 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1c6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af58a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094522:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000950ab880/0x4605c7c97e1bac22 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x14e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ace2a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094532:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000000fa8322f/0x4605c7c97e1bb926 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1c5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af536 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094539:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003ec13691/0x4605c7c97e1bb90a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1c4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af4e2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094545:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000c9ee59cd/0x4605c7c97e1bb8ee lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1c3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af48e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094555:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000812869c2/0x4605c7c97e1bb8d2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af43a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094560:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bca7647c/0x4605c7c97e1bac06 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x14d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acdd6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094566:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000002b9085/0x4605c7c97e1bb8b6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1c1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af3e6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094572:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b20b7c1f/0x4605c7c97e1bb0d6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x179:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adc46 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094578:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000009657afe5/0x4605c7c97e1bb87e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1bf:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af33e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094584:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000112482fa/0x4605c7c97e1bb0ba lrc: 0/0,0 mode: --/PW res: [0x240000400:0x178:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adbf2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094590:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000000c3c5ff/0x4605c7c97e1babea lrc: 0/0,0 mode: --/PW res: [0x240000400:0x14c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acd82 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094597:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f2dee099/0x4605c7c97e1bb846 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1bd:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af296 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094611:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e73ab13/0x4605c7c97e1bb09e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x177:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adb9e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094622:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000091ed0275/0x4605c7c97e1ba94a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x134:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac5a2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094632:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d55e254c/0x4605c7c97e1bb80e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1bb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af1ee expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094641:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000005081b4a/0x4605c7c97e1bb082 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x176:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adb4a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094648:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000007c73c2ab/0x4605c7c97e1babce lrc: 0/0,0 mode: --/PW res: [0x240000400:0x14b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acd2e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094654:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7e0281b/0x4605c7c97e1bb7d6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1b9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af146 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094660:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d26602ab/0x4605c7c97e1bb066 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x175:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4adaf6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094666:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000011062e6d/0x4605c7c97e1ba6c6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x11d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abe16 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094675:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000001a43a16/0x4605c7c97e1bb79e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1b7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af09e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094681:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000005c3c7b72/0x4605c7c97e1bb782 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1b6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4af04a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094687:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008e47f008/0x4605c7c97e1bb766 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1b5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aeff6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094692:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003c3df6c1/0x4605c7c97e1bb02e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x173:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ada4e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094698:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d78702d0/0x4605c7c97e1bb74a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1b4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aefa2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094705:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000014b58190/0x4605c7c97e1ba92e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x133:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac54e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094711:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a21d60e7/0x4605c7c97e1bb712 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1b2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aeefa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094716:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000adeec58e/0x4605c7c97e1bab96 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x149:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acc86 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094728:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000079f9bf6b/0x4605c7c97e1bb6f6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1b1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aeea6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094738:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000629b598a/0x4605c7c97e1baff6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x171:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad9a6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094748:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000caed67bb/0x4605c7c97e1bb6da lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1b0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aee52 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094758:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bcb29ae6/0x4605c7c97e1bb6be lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1af:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aedfe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094767:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000930555ec/0x4605c7c97e1bb6a2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1ae:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aedaa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094777:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000868327c9/0x4605c7c97e1bb686 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1ad:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aed56 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094787:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002b52a7c0/0x4605c7c97e1bb66a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1ac:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aed02 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094797:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f81291be/0x4605c7c97e1bb64e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1ab:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aecae expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094812:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000038b47fd8/0x4605c7c97e1bb632 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1aa:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aec5a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094822:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000077ac7f47/0x4605c7c97e1bb616 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1a9:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aec06 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094833:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003b08e68c/0x4605c7c97e1bb5fa lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1a8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aebb2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094855:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b2e0f464/0x4605c7c97e1bb5de lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1a7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aeb5e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094865:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000eed9ef14/0x4605c7c97e1bb5c2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1a6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aeb0a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094894:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7bf1831/0x4605c7c97e1bb5a6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1a5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aeab6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094906:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000057997c2e/0x4605c7c97e1bb58a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1a4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aea62 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094915:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006a8b96dd/0x4605c7c97e1bb56e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1a3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aea0e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094930:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000016ff7546/0x4605c7c97e1bb552 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1a2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae9ba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094940:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000013a04ab/0x4605c7c97e1bb536 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1a1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae966 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094949:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000053feb2e1/0x4605c7c97e1bb51a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x1a0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae912 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094959:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000b7fa819a/0x4605c7c97e1bb4fe lrc: 0/0,0 mode: --/PW res: [0x240000400:0x19f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae8be expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094969:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000024d6ae0c/0x4605c7c97e1bab0a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x144:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4acae2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094979:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000defa51e3/0x4605c7c97e1baec2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x166:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad60a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094989:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000956a613b/0x4605c7c97e1bae8a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x164:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad562 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.094999:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000089b2da7a/0x4605c7c97e1baa9a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x140:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac992 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095014:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000017ada4a3/0x4605c7c97e1baa7e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x13f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac93e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095024:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000008a85cb9c/0x4605c7c97e1baa62 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x13e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac8ea expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095034:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d1233094/0x4605c7c97e1badfe lrc: 0/0,0 mode: --/PW res: [0x240000400:0x15f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad3be expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095044:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000393e86fb/0x4605c7c97e1bb4aa lrc: 0/0,0 mode: --/PW res: [0x240000400:0x19c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae7c2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095055:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000354715db/0x4605c7c97e1baa46 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x13d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac896 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095066:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000094b0e1c1/0x4605c7c97e1badc6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x15d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad316 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095076:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e9ea91ce/0x4605c7c97e1ba86a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x12c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac302 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095086:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000068e6d36a/0x4605c7c97e1badaa lrc: 0/0,0 mode: --/PW res: [0x240000400:0x15c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad2c2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095101:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d6d88ce5/0x4605c7c97e1baa2a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x13c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac842 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095111:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000bde6697d/0x4605c7c97e1bad8e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x15b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad26e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095121:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000efd01301/0x4605c7c97e1ba68e lrc: 0/0,0 mode: --/PW res: [0x240000400:0x11b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abd6e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095132:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000002ce081f1/0x4605c7c97e1bad56 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x159:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad1c6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095142:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001e11d290/0x4605c7c97e1bb35a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x190:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ae3d2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095151:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000003b838344/0x4605c7c97e1baede lrc: 0/0,0 mode: --/PW res: [0x240000400:0x167:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad65e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095162:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000001f56ffcf/0x4605c7c97e1baaee lrc: 0/0,0 mode: --/PW res: [0x240000400:0x143:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aca8e expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095172:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000a527bd73/0x4605c7c97e1ba8be lrc: 0/0,0 mode: --/PW res: [0x240000400:0x12f:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac3fe expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095187:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000f066339b/0x4605c7c97e1baab6 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x141:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac9e6 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095197:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000e0bca7d7/0x4605c7c97e1ba752 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x122:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abfba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095207:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000030e06fa3/0x4605c7c97e1ba8a2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x12e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ac3aa expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095217:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 000000006b444f29/0x4605c7c97e1bae52 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x162:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad4ba expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095227:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000ebe9b562/0x4605c7c97e1bae36 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x161:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad466 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095237:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000022e395f3/0x4605c7c97e1bae1a lrc: 0/0,0 mode: --/PW res: [0x240000400:0x160:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4ad412 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095248:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 0000000029f229e5/0x4605c7c97e1ba6aa lrc: 0/0,0 mode: --/PW res: [0x240000400:0x11c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4abdc2 expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095258:0:5598:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff902350a11000 lock: 00000000d91a432d/0x4605c7c97e1baad2 lrc: 0/0,0 mode: --/PW res: [0x240000400:0x142:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400000020 nid: local remote: 0x9383f985bc4aca3a expref: -99 pid: 11448 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201702.095276:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201702.095278:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201702.095280:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201702.273089:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442801280:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201702.273213:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:0.0:1788201702.273221:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442801152:192.168.201.130@tcp:400:kworker.0 00000100:00100000:0.0:1788201702.273258:0:2359:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442801408:192.168.201.130@tcp:400:kworker.0 00010000:00010000:1.0:1788201703.105232:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201704.129027:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201705.153047:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201706.177035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201707.201044:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1788201707.264992:0:2360:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201691/real 1788201691] req@ffff90234519d180 x1875064442800000/t0(0) o103->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:17/18 lens 1632/224 e 0 to 1 dl 1788201707 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201707.274470:0:2360:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:1.0:1788201707.274478:0:2360:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff902350a11000: Connection to lustre-OST0000 (at 192.168.201.130@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1788201707.274483:0:2360:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00080000:3.0:1788201707.274586:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201707.274591:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201707.274594:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201707.274600:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 346 00000100:00080000:3.0:1788201707.274612:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201707.274620:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201707.274634:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442801792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201707.274677:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442801792:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788201707.274691:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442801536:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201707.274708:0:2362:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442801664:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201707.274779:0:2361:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442801920:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201708.225054:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788201708.288974:0:2362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201692/real 1788201692] req@ffff90235268a680 x1875064442800256/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201708 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201708.289007:0:2362:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff90235268a680 x1875064442800256/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201708 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201708.289014:0:2362:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1788201708.289017:0:2362:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff902350a11000: Connection to lustre-MDT0000 (at 192.168.201.130@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788201708.289022:0:2362:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00000400:2.0:1788201708.289044:0:2362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201692/real 1788201692] req@ffff902352689c00 x1875064442800128/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201708 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201708.289066:0:2362:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff902352689c00 x1875064442800128/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201708 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201708.289073:0:2362:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from FULL to DISCONN 00000100:02020000:2.0:1788201708.289076:0:2362:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.201.130@tcp: Connection to MGS (at 192.168.201.130@tcp) was lost; in progress operations using this service will fail 00000100:00080000:2.0:1788201708.292981:0:2362:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.201.130@tcp for MGC192.168.201.130@tcp not replayable, auto-deactivating 00000100:00080000:2.0:1788201708.292985:0:2362:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:2.0:1788201708.292991:0:2362:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff902352689c00 x1875064442800128/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201708 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201708.293004:0:2362:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff90234519e300 x1875064442800512/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201713 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201708.293019:0:2362:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff90234519f800 x1875064442801152/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201718 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201708.293028:0:2362:0:(client.c:3576:ptlrpc_abort_inflight()) @@@ inflight req@ffff902352688e00 x1875064442801536/t0(0) o400->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201723 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:2.0:1788201708.293053:0:2362:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1788201708.293056:0:2362:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1788201708.293058:0:2362:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:2.0:1788201708.293060:0:2362:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:0.0:1788201708.293071:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442801152:192.168.201.130@tcp:400:kworker.0 00000100:00100000:3.0:1788201708.293077:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7ad49433-54fa-4e21-b756-859420d95603:2361:1875064442801536:192.168.201.130@tcp:400:kworker.0 00000100:00000400:2.0:1788201708.293079:0:2362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201692/real 1788201692] req@ffff90235268b800 x1875064442800384/t0(0) o400->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201708 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201708.293082:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201708.293092:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1788201708.293096:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:2359:1875064442800512:192.168.201.130@tcp:400:kworker.0 00000100:00080000:1.0:1788201708.293098:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788201708.293108:0:2362:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff90235268b800 x1875064442800384/t0(0) o400->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201708 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201708.293109:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 937 00000100:00080000:2.0:1788201708.293120:0:2362:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from FULL to DISCONN 00000100:00100000:0.0:1788201708.293123:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7ad49433-54fa-4e21-b756-859420d95603:0:1875064442802048:192.168.201.130@tcp:400:kworker.0 00000100:00080000:1.0:1788201708.293127:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:02000400:2.0:1788201708.293129:0:2362:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0001-osc-ffff902350a11000: Connection to lustre-OST0001 (at 192.168.201.130@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788201708.293138:0:2362:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0001_UUID: waking up pinger s:DISCONN 00000008:00100000:1.0:1788201708.293140:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201708.293160:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442802176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201708.293168:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442800256:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201708.293179:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7ad49433-54fa-4e21-b756-859420d95603:2362:1875064442800128:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201708.293184:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442800384:192.168.201.130@tcp:400:kworker.0 00000100:00080000:1.0:1788201708.293190:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201708.293195:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201708.293199:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201708.293203:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201708.293204:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201708.293209:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 944 00000100:00080000:1.0:1788201708.293214:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201708.293219:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201708.293230:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442802304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201708.293242:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201708.293245:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201708.293247:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201708.293251:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 927 00000100:00080000:1.0:1788201708.293255:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788201708.293264:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442802432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201708.293269:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442802176:192.168.201.130@tcp:38:kworker.0 00000100:00080000:1.0:1788201708.293282:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201708.293286:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788201708.293354:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442802304:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788201708.293387:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442802432:192.168.201.130@tcp:250:kworker.0 00010000:00010000:2.0:1788201709.249047:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201710.273023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201711.297015:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201712.320974:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201712.320993:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201712.320997:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201712.321001:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201712.321004:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201712.321008:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201712.321012:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201712.321015:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1788201712.321022:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201713.344970:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201713.344980:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201713.344984:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201713.344988:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1788201713.344998:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201713.345001:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201713.345006:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201713.345009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201713.345013:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:0.0:1788201713.408974:0:2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201697/real 1788201697] req@ffff90234519d880 x1875064442800768/t0(0) o400->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201713 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788201713.408974:0:2360:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201697/real 1788201697] req@ffff90234519dc00 x1875064442800640/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201713 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201713.409021:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff90234519d880 x1875064442800768/t0(0) o400->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201713 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201713.409027:0:2360:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff90234519dc00 x1875064442800640/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201713 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201713.409033:0:2359:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff902348893000 already not connected (conn 3, was 2): CONNECTING 00000100:00080000:1.0:1788201713.409042:0:2360:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 18, was 17): CONNECTING 00000100:00000400:1.0:1788201713.409048:0:2360:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201697/real 1788201697] req@ffff90234519fb80 x1875064442800896/t0(0) o400->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201713 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201713.409055:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442800768:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201713.409062:0:2360:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff90234519fb80 x1875064442800896/t0(0) o400->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201713 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201713.409067:0:2360:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff902348894000 already not connected (conn 6, was 5): CONNECTING 00000100:00100000:1.0:1788201713.409076:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442800640:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201713.409080:0:2360:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2360:1875064442800896:192.168.201.130@tcp:400:kworker.0 00010000:00010000:1.0:1788201714.369021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201715.393063:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201716.416984:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201717.440955:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201707/real 1788201707] req@ffff902352689500 x1875064442801792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201717 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201717.440990:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442801792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201717 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1788201717.441001:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201717.441012:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201717.441031:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201717.441035:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442801792:192.168.201.130@tcp:8:kworker.0 00000100:00080000:1.0:1788201717.441058:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201717.441064:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201717.441067:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201717.441070:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201717.441074:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201717.441077:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201717.441080:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201717.441086:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 983 00000100:00080000:1.0:1788201717.441095:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1788201717.441101:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201717.441112:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201717.441149:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442802560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201717.441170:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201717.441174:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1788201717.441207:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442802560:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788201718.464983:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201708/real 1788201708] req@ffff90234519ed80 x1875064442802432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1788201718.465012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201718.465020:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442802432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201718 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201718.465030:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201708/real 1788201708] req@ffff90234519df80 x1875064442802176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201718.465049:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442802176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201718 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201718.465056:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201708/real 1788201708] req@ffff90234519c700 x1875064442802304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201718.465063:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442802304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201718 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201718.465076:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201718.465094:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201718.465098:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442802432:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788201718.465103:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201718.465108:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201718.465111:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442802176:192.168.201.130@tcp:38:kworker.0 00000100:00080000:2.0:1788201718.465116:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201718.465117:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201718.465121:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201718.465124:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201718.465128:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00080000:1.0:1788201718.465129:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 984 00000100:00100000:2.0:1788201718.465132:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442802304:192.168.201.130@tcp:8:kworker.0 00000100:00080000:1.0:1788201718.465137:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1788201718.465143:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788201718.465175:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442802688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201718.465196:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201718.465199:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201718.465201:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201718.465205:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 984 00000100:00080000:1.0:1788201718.465208:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1788201718.465212:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201718.465220:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201718.465222:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442802688:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201718.465231:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442802816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201718.465239:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201718.465242:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201718.465244:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201718.465246:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201718.465248:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201718.465251:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 984 00000100:00080000:1.0:1788201718.465253:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1788201718.465256:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201718.465259:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201718.465267:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442802944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201718.465278:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201718.465281:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1788201718.465306:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442802816:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201718.465333:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442802944:192.168.201.130@tcp:8:kworker.0 00000100:00000400:1.0:1788201718.528997:0:2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201702/real 1788201702] req@ffff90234519c380 x1875064442801408/t0(0) o400->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201718.529030:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff90234519c380 x1875064442801408/t0(0) o400->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201718 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201718.529037:0:2359:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff902348894000 already not connected (conn 7, was 5): CONNECTING 00000100:00000400:1.0:1788201718.529042:0:2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201702/real 1788201702] req@ffff90234519f480 x1875064442801280/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201718.529050:0:2359:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff90234519f480 x1875064442801280/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201718 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201718.529054:0:2359:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 19, was 17): CONNECTING 00000100:00000400:1.0:1788201718.529057:0:2359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201702/real 1788201702] req@ffff90234519ca80 x1875064442801024/t0(0) o103->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:17/18 lens 2048/224 e 0 to 1 dl 1788201718 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201718.529063:0:2359:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff902348893000 already not connected (conn 4, was 2): CONNECTING 00000100:00100000:1.0:1788201718.529074:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442801408:192.168.201.130@tcp:400:kworker.0 00000100:00100000:1.0:1788201718.529078:0:2359:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2359:1875064442801280:192.168.201.130@tcp:400:kworker.0 00000080:00200000:2.0:1788201718.657088:0:38:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x42e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201718.657102:0:38:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00020000:00200000:2.0:1788201718.657119:0:38:0:(lov_io.c:1085:lov_io_iter_init()) component[1] flags 0x0 00000080:00200000:2.0:1788201718.657146:0:38:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x42e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1788201719.489029:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201720.513024:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201721.537022:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201722.496948:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201722.496956:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201722.496959:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201722.496961:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201722.496963:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201722.496965:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201722.496967:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201722.496968:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1788201722.561015:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201723.520972:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201723.520980:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201723.520983:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201723.520986:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201723.520988:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201723.520990:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201723.520991:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201723.520993:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1788201723.585024:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788201723.776984:0:2361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201707/real 1788201707] req@ffff902352688000 x1875064442801920/t0(0) o400->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:3.0:1788201723.776984:0:2362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201707/real 1788201707] req@ffff90235268b100 x1875064442801664/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201723.777033:0:2361:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff902352688000 x1875064442801920/t0(0) o400->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201723 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201723.777034:0:2362:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff90235268b100 x1875064442801664/t0(0) o400->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201723 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201723.777046:0:2362:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff902348897000 already not connected (conn 19, was 17): CONNECTING 00000100:00080000:2.0:1788201723.777057:0:2361:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff902348894000 already not connected (conn 7, was 5): CONNECTING 00000100:00100000:3.0:1788201723.777061:0:2362:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2362:1875064442801664:192.168.201.130@tcp:400:kworker.0 00000100:00100000:2.0:1788201723.777072:0:2361:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2361:1875064442801920:192.168.201.130@tcp:400:kworker.0 00010000:00010000:2.0:1788201724.609074:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201725.633071:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201726.657004:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201727.681031:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201728.705055:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201728.897010:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201728.897019:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201728.897021:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201728.897023:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201728.897024:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201728.897026:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201728.897028:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201728.897030:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1788201729.729096:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201730.753025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201731.777075:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201732.737041:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201717/real 1788201717] req@ffff90234519fb80 x1875064442802560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201732 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201732.737087:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442802560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201732 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201732.737114:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201732.737139:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201732.737143:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442802560:192.168.201.130@tcp:8:kworker.0 00000100:00080000:1.0:1788201732.737215:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201732.737225:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201732.737228:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201732.737232:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201732.737242:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201732.737248:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201732.737252:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201732.737260:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 994 00000100:00080000:1.0:1788201732.737271:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1788201732.737277:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201732.737288:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201732.737331:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442803072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201732.737359:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201732.737362:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1788201732.737410:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442803072:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201732.801028:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201733.760985:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201718/real 1788201718] req@ffff90234519ea00 x1875064442802816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201733 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201733.761013:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442802816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201733 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201733.761023:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201718/real 1788201718] req@ffff90234519f100 x1875064442802944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201733 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201733.761032:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442802944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201733 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201733.761040:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201718/real 1788201718] req@ffff90234519dc00 x1875064442802688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201733 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201733.761049:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442802688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201733 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201733.761066:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201733.761085:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201733.761089:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442802816:192.168.201.130@tcp:38:kworker.0 00000100:00080000:2.0:1788201733.761093:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201733.761097:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201733.761100:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442802944:192.168.201.130@tcp:8:kworker.0 00000100:00080000:2.0:1788201733.761104:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201733.761110:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201733.761113:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442802688:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788201733.761124:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201733.761130:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201733.761134:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201733.761140:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 995 00000100:00080000:1.0:1788201733.761150:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1788201733.761156:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788201733.761186:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442803200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201733.761205:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201733.761208:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201733.761209:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201733.761213:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 995 00000100:00080000:1.0:1788201733.761216:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1788201733.761219:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201733.761229:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201733.761234:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442803200:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201733.761242:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442803328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201733.761251:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201733.761254:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201733.761256:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201733.761259:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201733.761260:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201733.761264:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 995 00000100:00080000:1.0:1788201733.761267:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1788201733.761270:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201733.761274:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201733.761283:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442803456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201733.761334:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442803328:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201733.761361:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442803456:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201733.825028:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201734.848985:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201735.873049:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201736.897025:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201737.792982:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201737.792993:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201737.792997:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201737.792999:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201737.793003:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201737.793008:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201737.793011:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201737.793014:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1788201737.921061:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201738.817043:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201738.817050:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201738.817052:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201738.817055:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201738.817056:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201738.817058:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201738.817059:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201738.817061:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1788201738.945048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201739.969016:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201740.993013:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201742.017043:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201743.040994:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201744.065052:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201744.256959:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201744.256969:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201744.256972:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201744.256975:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201744.256978:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201744.256981:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201744.256984:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201744.256986:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1788201745.089130:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201746.113004:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201747.137076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.1F:1788201748.032954:0:2350:0:(socklnd_cb.c:2287:ksocknal_find_timed_out_conn()) Timeout sending data to 12345-192.168.201.130@tcp (192.168.201.130:988) the network or that node may be down. 00010000:00010000:1.0:1788201748.161024:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201749.185040:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201749.376985:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201749.376993:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201749.376996:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201749.377000:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201749.377003:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201749.377007:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201749.377009:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201749.377013:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1788201750.209056:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201751.233039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201752.129003:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201732/real 1788201732] req@ffff90234519f480 x1875064442803072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201752.129042:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442803072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201752 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201752.129064:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201752.129081:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201752.129086:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442803072:192.168.201.130@tcp:8:kworker.0 00000100:00080000:2.0:1788201752.129112:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201752.129118:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201752.129121:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201752.129124:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201752.129127:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201752.129130:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201752.129134:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201752.129140:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1009 00000100:00080000:2.0:1788201752.129150:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1788201752.129157:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788201752.129168:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201752.129198:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442803584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201752.129219:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201752.129223:0:11804:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788201752.129264:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442803584:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201752.257060:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201753.152973:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201733/real 1788201733] req@ffff90234519dc00 x1875064442803328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201753 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201753.152999:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442803328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201753 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201753.153009:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201733/real 1788201733] req@ffff90234519f100 x1875064442803456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201753 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201753.153018:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442803456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201753 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201753.153025:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201733/real 1788201733] req@ffff90234519c380 x1875064442803200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201753 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201753.153034:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442803200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201753 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201753.153054:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201753.153073:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201753.153077:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442803328:192.168.201.130@tcp:38:kworker.0 00000100:00080000:1.0:1788201753.153082:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201753.153102:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00080000:3.0:1788201753.153105:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788201753.153109:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442803456:192.168.201.130@tcp:8:kworker.0 00000100:00080000:3.0:1788201753.153111:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788201753.153118:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201753.153123:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201753.153123:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201753.153133:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201753.153139:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00080000:3.0:1788201753.153141:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1010 00000100:00100000:1.0:1788201753.153147:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442803200:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201753.153151:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788201753.153161:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201753.153170:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201753.153196:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442803712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201753.153215:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201753.153218:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788201753.153221:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201753.153223:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201753.153224:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000400:00000100:0.0:1788201753.153226:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.130:988 00000100:00080000:3.0:1788201753.153227:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1010 00000100:00080000:3.0:1788201753.153234:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788201753.153240:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201753.153245:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201753.153255:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442803840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201753.153264:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442803712:192.168.201.130@tcp:38:kworker.0 00000400:00000100:0.0:1788201753.153267:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000100:00080000:3.0:1788201753.153269:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201753.153275:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201753.153277:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201753.153280:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1010 00000100:00080000:3.0:1788201753.153284:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788201753.153286:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:0.0:1788201753.153296:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788201753.153305:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442803968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201753.153314:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201753.153318:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000800:00000100:1.0:1788201753.153331:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201753.153365:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201753/real 1788201753] req@ffff90235268b100 x1875064442803712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201753.153380:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442803712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201753.153388:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201753.153392:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201753.153395:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442803712:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201753.153401:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201752/real 1788201753] req@ffff90234598ce00 x1875064442803584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201753.153418:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442803584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201753.153426:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201753.153429:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201753.153433:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442803584:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201753.153451:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442803840:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201753.153475:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201753.153489:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201753/real 1788201753] req@ffff902352688e00 x1875064442803840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201753.153499:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442803840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201753.153508:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201753.153511:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201753.153514:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442803840:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201753.153521:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442803968:192.168.201.130@tcp:250:kworker.0 00000800:00000100:1.0:1788201753.153556:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201753.153571:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201753/real 1788201753] req@ffff90235268a300 x1875064442803968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201753.153583:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442803968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201753.153597:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201753.153600:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201753.153604:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442803968:192.168.201.130@tcp:250:kworker.0 00010000:00010000:2.0:1788201753.281027:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201754.305016:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201755.329059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201756.353001:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201757.184974:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201757.184982:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201757.184984:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201757.184993:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1029 00000100:00080000:3.0:1788201757.185002:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201757.185008:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201757.185042:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442804096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201757.185068:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201757.185071:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201757.185073:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201757.185076:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1029 00000100:00080000:3.0:1788201757.185079:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201757.185081:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201757.185088:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201757.185096:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442804224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201757.185104:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201757.185107:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201757.185113:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201757.185117:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1028 00000100:00080000:3.0:1788201757.185119:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201757.185121:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201757.185126:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201757.185127:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442804096:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788201757.185137:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442804352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201757.185145:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201757.185148:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201757.185149:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201757.185151:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1029 00000100:00080000:3.0:1788201757.185153:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201757.185156:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201757.185159:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201757.185167:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442804480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201757.185227:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442804224:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201757.185270:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442804352:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201757.185299:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442804480:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201757.377034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201758.401001:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788201759.297164:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.130:988 00000400:00000100:0.0:1788201759.297201:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788201759.297220:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201759.297252:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201759.297262:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201759.297271:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:1.0:1788201759.297284:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201757/real 1788201759] req@ffff90235268b100 x1875064442804352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201759.297311:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442804352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201759.297332:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201759.297336:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201759.297341:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442804352:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201759.297350:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201757/real 1788201759] req@ffff902352688000 x1875064442804480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201759.297373:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442804480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201759.297380:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201759.297383:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201759.297387:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442804480:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201759.297394:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201757/real 1788201759] req@ffff90235268a300 x1875064442804096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201759.297408:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442804096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201759.297416:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201759.297420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201759.297423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442804096:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201759.297430:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201757/real 1788201759] req@ffff902352688e00 x1875064442804224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201759.297442:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442804224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201759.297450:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201759.297454:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201759.297458:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442804224:192.168.201.130@tcp:38:kworker.0 00010000:00010000:1.0:1788201759.424989:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201760.449061:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201761.473009:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201762.497080:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201762.689025:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201762.689041:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201762.689046:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201762.689058:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1033 00000100:00080000:3.0:1788201762.689073:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201762.689081:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201762.689131:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442804608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201762.689171:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201762.689176:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201762.689178:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201762.689182:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1033 00000100:00080000:3.0:1788201762.689186:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201762.689191:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201762.689202:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201762.689217:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442804736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201762.689228:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442804608:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201762.689231:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201762.689238:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201762.689240:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201762.689245:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1033 00000100:00080000:3.0:1788201762.689249:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201762.689254:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201762.689259:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201762.689272:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442804864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201762.689283:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201762.689286:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201762.689288:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201762.689293:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1033 00000100:00080000:3.0:1788201762.689296:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201762.689301:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201762.689305:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201762.689316:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442804992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201762.689353:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442804736:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201762.689378:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442804864:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201762.689394:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442804992:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201763.520997:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201764.545068:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788201765.441170:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.130:988 00000400:00000100:0.0:1788201765.441209:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788201765.441220:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201765.441251:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201765.441258:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201765.441263:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:1.0:1788201765.441301:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201762/real 1788201765] req@ffff90235268b800 x1875064442804736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201765.441325:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442804736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201765.441336:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201765.441340:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201765.441344:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442804736:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201765.441352:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201762/real 1788201765] req@ffff902352689c00 x1875064442804864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201765.441362:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442804864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201765.441370:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201765.441373:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201765.441376:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442804864:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201765.441383:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201762/real 1788201765] req@ffff90235268a680 x1875064442804992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201765.441393:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442804992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201765.441399:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201765.441402:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201765.441410:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442804992:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201765.441415:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201762/real 1788201765] req@ffff902352689500 x1875064442804608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201765.441429:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442804608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201765.441436:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201765.441439:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201765.441442:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442804608:192.168.201.130@tcp:250:kworker.0 00010000:00010000:2.0:1788201765.568994:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201766.593009:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201767.617019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201767.808940:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201767.808951:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201767.808955:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201767.808966:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1039 00000100:00080000:3.0:1788201767.808977:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201767.808985:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201767.809023:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442805120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201767.809062:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201767.809069:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201767.809071:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201767.809076:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1039 00000100:00080000:3.0:1788201767.809080:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201767.809084:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201767.809092:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201767.809103:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442805248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201767.809113:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442805120:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201767.809113:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201767.809121:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201767.809124:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201767.809128:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1039 00000100:00080000:3.0:1788201767.809130:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201767.809132:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201767.809136:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201767.809143:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442805376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201767.809150:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201767.809153:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201767.809154:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201767.809156:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1039 00000100:00080000:3.0:1788201767.809159:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201767.809161:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201767.809164:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201767.809172:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442805504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201767.809183:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201767.809225:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201767/real 1788201767] req@ffff902352688700 x1875064442805120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201767.809241:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442805120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201767.809253:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201767.809255:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201767.809258:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442805120:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201767.809275:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442805248:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201767.809291:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201767.809300:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201767/real 1788201767] req@ffff902352689180 x1875064442805248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201767.809308:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442805248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201767.809313:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201767.809315:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201767.809317:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442805248:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201767.809320:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442805376:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201767.809331:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201767.809337:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201767/real 1788201767] req@ffff902352689880 x1875064442805376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201767.809344:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442805376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201767.809350:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201767.809352:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201767.809354:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442805376:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201767.809358:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442805504:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201767.809368:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201767.809374:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201767/real 1788201767] req@ffff902352688380 x1875064442805504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201767.809383:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442805504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201767.809391:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201767.809394:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201767.809397:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442805504:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201768.641026:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201769.665051:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201770.689004:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201771.713017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201772.737015:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201772.929016:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201772.929025:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201772.929028:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201772.929037:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1044 00000100:00080000:0.0:1788201772.929046:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201772.929052:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788201772.929089:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442805632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201772.929120:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201772.929124:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201772.929126:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201772.929131:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1044 00000100:00080000:0.0:1788201772.929135:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201772.929138:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201772.929147:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201772.929159:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442805760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201772.929167:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201772.929170:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201772.929172:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201772.929176:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1044 00000100:00080000:0.0:1788201772.929179:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201772.929189:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788201772.929193:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442805632:192.168.201.130@tcp:250:kworker.0 00000008:00100000:0.0:1788201772.929196:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201772.929217:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442805888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201772.929231:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201772.929234:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201772.929235:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201772.929238:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1044 00000100:00080000:0.0:1788201772.929241:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201772.929244:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201772.929248:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201772.929258:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442806016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201772.929340:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442805760:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201772.929394:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442805888:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201772.929412:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442806016:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201773.761023:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788201774.657235:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.130:988 00000400:00000100:0.0:1788201774.657281:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788201774.657296:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201774.657335:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201774.657345:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201774.657355:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:1.0:1788201774.657386:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201772/real 1788201774] req@ffff9023450bb800 x1875064442805888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201774.657425:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442805888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201774.657456:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201774.657461:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201774.657466:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442805888:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201774.657474:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201772/real 1788201774] req@ffff9023450b8a80 x1875064442806016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201774.657490:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442806016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201774.657498:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201774.657502:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201774.657506:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442806016:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201774.657514:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201772/real 1788201774] req@ffff9023450b9500 x1875064442805632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201774.657527:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442805632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201774.657536:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201774.657539:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201774.657543:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442805632:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201774.657550:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201772/real 1788201774] req@ffff9023450ba680 x1875064442805760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201774.657561:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442805760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201774.657569:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201774.657572:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201774.657586:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442805760:192.168.201.130@tcp:38:kworker.0 00010000:00010000:1.0:1788201774.785041:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201775.809037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201776.833080:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201777.857036:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201778.049026:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201778.049037:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201778.049041:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201778.049052:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1049 00000100:00080000:3.0:1788201778.049066:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201778.049074:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201778.049119:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442806144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201778.049147:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201778.049158:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201778.049164:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201778.049170:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1049 00000100:00080000:3.0:1788201778.049174:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201778.049180:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201778.049193:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201778.049207:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442806272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201778.049213:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442806144:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201778.049219:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201778.049224:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201778.049226:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201778.049229:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1049 00000100:00080000:3.0:1788201778.049233:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201778.049237:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201778.049241:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201778.049251:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442806400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201778.049258:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201778.049262:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201778.049264:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201778.049269:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1049 00000100:00080000:3.0:1788201778.049273:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201778.049277:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201778.049283:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201778.049296:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064442806528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201778.049313:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201778.049373:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201778/real 1788201778] req@ffff90235268bb80 x1875064442806144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201778.049399:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442806144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201778.049409:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201778.049412:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201778.049416:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442806144:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201778.049439:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442806272:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201778.049458:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201778.049468:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201778/real 1788201778] req@ffff90235268b480 x1875064442806272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201778.049476:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442806272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201778.049486:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201778.049489:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201778.049491:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442806272:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201778.049495:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442806400:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201778.049506:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201778.049534:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201778/real 1788201778] req@ffff902352688a80 x1875064442806400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201778.049542:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442806400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201778.049549:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201778.049551:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201778.049553:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442806400:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201778.049557:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442806528:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201778.049569:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201778.049575:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201778/real 1788201778] req@ffff90235268aa00 x1875064442806528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201778.049582:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064442806528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201778.049587:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201778.049589:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201778.049592:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442806528:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201778.881006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201779.905050:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201780.929019:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201781.953018:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201782.977021:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201783.168958:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201783.168970:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201783.168974:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201783.168985:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1054 00000100:00080000:3.0:1788201783.168998:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201783.169005:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201783.169052:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442806656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201783.169080:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201783.169085:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201783.169087:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201783.169090:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1054 00000100:00080000:3.0:1788201783.169093:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201783.169095:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201783.169105:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201783.169117:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064442806784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201783.169129:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788201783.169132:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442806656:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201783.169134:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201783.169138:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201783.169145:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1054 00000100:00080000:3.0:1788201783.169150:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201783.169155:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201783.169161:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201783.169175:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442806912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201783.169183:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201783.169186:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201783.169188:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201783.169192:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1054 00000100:00080000:3.0:1788201783.169197:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201783.169201:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201783.169208:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201783.169221:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442807040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201783.169260:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442806784:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201783.169291:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442806912:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201783.169312:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442807040:192.168.201.130@tcp:8:kworker.0 00000400:00000100:0.0:1788201783.873208:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.130:988 00000400:00000100:0.0:1788201783.873269:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788201783.873301:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201783.873354:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201783.873384:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201783.873407:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:1.0:1788201783.873413:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201783/real 1788201783] req@ffff90235268aa00 x1875064442806784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201783.873469:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064442806784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201783.873486:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201783.873492:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201783.873497:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442806784:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201783.873507:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201783/real 1788201783] req@ffff902352688a80 x1875064442806912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201783.873519:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442806912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201783.873526:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201783.873529:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201783.873532:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442806912:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201783.873548:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201783/real 1788201783] req@ffff90235268b480 x1875064442807040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201783.873562:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442807040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201783.873571:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201783.873574:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201783.873578:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442807040:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201783.873586:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201783/real 1788201783] req@ffff90235268ad80 x1875064442806656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201783.873598:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442806656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201783.873607:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201783.873620:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201783.873623:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442806656:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788201784.001075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201785.025083:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201786.049053:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201787.073036:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201788.097037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201788.288992:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201788.289004:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201788.289008:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201788.289019:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1059 00000100:00080000:3.0:1788201788.289032:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201788.289039:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201788.289085:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442807168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201788.289113:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201788.289116:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201788.289117:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201788.289121:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1059 00000100:00080000:3.0:1788201788.289123:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201788.289126:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201788.289136:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201788.289151:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442807296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201788.289164:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788201788.289168:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442807168:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201788.289170:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201788.289175:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201788.289181:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1059 00000100:00080000:3.0:1788201788.289187:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201788.289192:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201788.289198:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201788.289212:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442807424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201788.289220:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201788.289223:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201788.289225:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201788.289228:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1059 00000100:00080000:3.0:1788201788.289241:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201788.289246:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:1.0:1788201788.289250:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:3.0:1788201788.289255:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201788.289272:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442807552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201788.289299:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201788/real 1788201788] req@ffff90235268bb80 x1875064442807168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201788.289317:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442807168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201788.289325:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201788.289329:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201788.289332:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442807168:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201788.289379:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442807296:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201788.289398:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201788.289408:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201788/real 1788201788] req@ffff902352688380 x1875064442807296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201788.289417:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442807296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201788.289423:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201788.289425:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201788.289427:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442807296:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201788.289431:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442807424:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201788.289442:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201788.289448:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201788/real 1788201788] req@ffff902352689880 x1875064442807424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201788.289456:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442807424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201788.289466:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201788.289469:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201788.289471:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442807424:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201788.289474:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442807552:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201788.289484:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201788.289490:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201788/real 1788201788] req@ffff902352689180 x1875064442807552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201788.289496:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442807552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201788.289501:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201788.289503:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201788.289505:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442807552:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201789.121010:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201790.145041:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201791.169051:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201792.193060:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201793.217035:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201793.409007:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201793.409015:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201793.409019:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201793.409028:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1064 00000100:00080000:0.0:1788201793.409038:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201793.409043:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788201793.409076:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442807680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201793.409098:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201793.409101:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201793.409103:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201793.409105:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1064 00000100:00080000:0.0:1788201793.409108:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201793.409111:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201793.409119:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201793.409128:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442807808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201793.409135:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201793.409138:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201793.409139:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201793.409141:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1064 00000100:00080000:0.0:1788201793.409143:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201793.409148:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788201793.409150:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442807680:192.168.201.130@tcp:250:kworker.0 00000008:00100000:0.0:1788201793.409152:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201793.409163:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442807936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201793.409171:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201793.409174:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201793.409175:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201793.409178:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1064 00000100:00080000:0.0:1788201793.409180:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201793.409183:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201793.409197:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201793.409207:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442808064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201793.409212:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201793.409270:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201793/real 1788201793] req@ffff9023450bb480 x1875064442807680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201793.409292:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442807680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201793.409310:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201793.409313:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201793.409317:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442807680:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201793.409322:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442807808:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201793.409341:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201793.409351:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201793/real 1788201793] req@ffff9023450bbb80 x1875064442807808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201793.409360:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442807808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201793.409365:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201793.409368:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201793.409370:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442807808:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201793.409388:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442807936:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201793.409399:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201793.409406:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201793/real 1788201793] req@ffff9023450b8e00 x1875064442807936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201793.409413:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442807936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201793.409419:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201793.409420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201793.409422:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442807936:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201793.409426:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442808064:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201793.409435:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201793.409441:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201793/real 1788201793] req@ffff9023450ba680 x1875064442808064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201793.409449:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442808064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201793.409454:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201793.409455:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201793.409457:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442808064:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788201794.241101:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201795.265026:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201796.289062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201797.313045:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201798.337044:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201798.528997:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201798.529009:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201798.529014:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201798.529028:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1070 00000100:00080000:0.0:1788201798.529043:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201798.529050:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788201798.529095:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442808192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201798.529119:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201798.529122:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201798.529124:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201798.529127:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1070 00000100:00080000:0.0:1788201798.529129:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201798.529132:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201798.529141:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201798.529156:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442808320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201798.529181:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201798.529184:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201798.529188:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788201798.529188:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442808192:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788201798.529195:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1070 00000100:00080000:0.0:1788201798.529199:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201798.529203:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201798.529208:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201798.529220:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442808448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201798.529227:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201798.529230:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201798.529231:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201798.529234:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1070 00000100:00080000:0.0:1788201798.529248:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201798.529251:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201798.529255:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788201798.529264:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201798.529269:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442808576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201798.529308:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201798/real 1788201798] req@ffff9023450b9500 x1875064442808192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201798.529326:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442808192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201798.529334:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201798.529346:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201798.529351:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442808192:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201798.529378:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442808320:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201798.529405:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201798.529420:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201798/real 1788201798] req@ffff9023450b8a80 x1875064442808320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201798.529445:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442808320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201798.529454:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201798.529457:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201798.529460:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442808320:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201798.529465:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442808448:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201798.529489:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201798.529507:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201798/real 1788201798] req@ffff9023450bb800 x1875064442808448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201798.529523:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442808448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201798.529533:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201798.529538:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201798.529544:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442808448:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201798.529554:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442808576:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201798.529579:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201798.529591:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201798/real 1788201798] req@ffff9023450b8380 x1875064442808576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201798.529605:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442808576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201798.529613:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201798.529616:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201798.529618:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442808576:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201799.361019:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201800.385027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201801.409042:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201802.433013:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201803.457149:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201803.648998:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201803.649006:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201803.649010:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201803.649018:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1075 00000100:00080000:1.0:1788201803.649027:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201803.649032:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788201803.649084:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442808704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201803.649109:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201803.649112:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201803.649113:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201803.649117:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1075 00000100:00080000:1.0:1788201803.649120:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201803.649123:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201803.649132:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201803.649143:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442808832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201803.649153:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788201803.649154:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442808704:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788201803.649159:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201803.649162:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201803.649167:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1075 00000100:00080000:1.0:1788201803.649169:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201803.649171:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201803.649174:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201803.649184:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442808960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201803.649190:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201803.649193:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201803.649194:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201803.649196:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1075 00000100:00080000:1.0:1788201803.649198:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201803.649200:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201803.649202:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201803.649208:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442809088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201803.649316:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442808832:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788201803.649339:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442808960:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788201803.649355:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442809088:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201804.481003:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788201805.377240:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:1.0:1788201805.377294:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788201805.377311:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788201805.377361:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788201805.377378:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788201805.377390:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788201805.377432:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201803/real 1788201805] req@ffff90234519f100 x1875064442808832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201805.377487:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442808832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201805.377510:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201805.377517:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201805.377522:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442808832:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788201805.377532:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201803/real 1788201805] req@ffff90234519dc00 x1875064442808960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201805.377563:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442808960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201805.377573:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201805.377577:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201805.377582:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442808960:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788201805.377591:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201803/real 1788201805] req@ffff90234519f480 x1875064442809088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201805.377604:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442809088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201805.377619:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201805.377624:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201805.377631:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442809088:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788201805.377640:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201803/real 1788201805] req@ffff90234519c380 x1875064442808704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201805.377656:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442808704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201805.377666:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201805.377670:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201805.377674:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442808704:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788201805.505069:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201806.529094:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201807.553068:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201808.577057:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201808.768972:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201808.768981:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201808.769003:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201808.769015:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1080 00000100:00080000:3.0:1788201808.769026:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201808.769031:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201808.769064:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442809216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201808.769088:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201808.769094:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201808.769096:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201808.769101:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1080 00000100:00080000:3.0:1788201808.769106:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201808.769111:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201808.769120:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201808.769128:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442809216:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788201808.769137:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442809344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201808.769151:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201808.769154:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201808.769155:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201808.769158:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1080 00000100:00080000:3.0:1788201808.769161:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201808.769164:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201808.769167:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201808.769177:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442809472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201808.769183:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201808.769186:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201808.769187:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201808.769190:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1080 00000100:00080000:3.0:1788201808.769192:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201808.769197:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:0.0:1788201808.769198:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:3.0:1788201808.769203:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201808.769211:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442809600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201808.769239:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201808/real 1788201808] req@ffff902352688700 x1875064442809216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201808.769259:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442809216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201808.769270:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201808.769274:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201808.769278:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442809216:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788201808.769301:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442809344:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788201808.769321:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201808.769331:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201808/real 1788201808] req@ffff902352689500 x1875064442809344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201808.769348:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442809344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201808.769354:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201808.769357:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201808.769359:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442809344:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788201808.769364:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442809472:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788201808.769377:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201808.769384:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201808/real 1788201808] req@ffff90235268a680 x1875064442809472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201808.769392:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442809472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201808.769397:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201808.769399:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201808.769401:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442809472:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788201808.769405:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442809600:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788201808.769415:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201808.769421:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201808/real 1788201808] req@ffff902352689c00 x1875064442809600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201808.769433:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442809600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201808.769441:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201808.769444:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201808.769447:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442809600:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201809.601053:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201810.625026:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201811.649029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201812.673052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201813.696990:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201813.888981:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201813.888989:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201813.888992:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201813.889001:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1085 00000100:00080000:3.0:1788201813.889010:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201813.889016:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201813.889046:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442809728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201813.889070:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201813.889073:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201813.889074:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201813.889081:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1085 00000100:00080000:3.0:1788201813.889083:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201813.889086:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201813.889094:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201813.889103:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442809856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201813.889111:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201813.889113:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201813.889114:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201813.889117:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1085 00000100:00080000:3.0:1788201813.889120:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201813.889123:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201813.889126:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201813.889136:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442809728:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788201813.889136:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442809984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201813.889151:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201813.889154:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201813.889155:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201813.889158:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1085 00000100:00080000:3.0:1788201813.889159:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201813.889162:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201813.889164:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201813.889171:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442810112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788201813.889227:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201813.889264:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201813/real 1788201813] req@ffff90235268b800 x1875064442809728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201813.889278:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442809728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201813.889286:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201813.889289:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201813.889292:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442809728:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788201813.889297:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442809856:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788201813.889312:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201813.889321:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201813/real 1788201813] req@ffff902352688e00 x1875064442809856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201813.889335:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442809856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201813.889341:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201813.889343:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201813.889345:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442809856:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788201813.889364:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442809984:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788201813.889376:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201813.889383:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201813/real 1788201813] req@ffff90235268a300 x1875064442809984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201813.889390:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442809984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201813.889399:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201813.889401:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201813.889403:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442809984:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788201813.889407:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442810112:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788201813.889416:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201813.889421:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201813/real 1788201813] req@ffff902352688000 x1875064442810112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201813.889428:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442810112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201813.889433:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201813.889435:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201813.889436:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442810112:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201814.721035:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201815.745087:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201816.769075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201817.793068:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201818.817062:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201819.008984:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201819.008992:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201819.008995:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201819.009005:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1090 00000100:00080000:3.0:1788201819.009016:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201819.009023:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201819.009061:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442810240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201819.009086:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201819.009091:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201819.009092:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201819.009096:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1090 00000100:00080000:3.0:1788201819.009099:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201819.009102:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201819.009110:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201819.009120:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442810368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201819.009127:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201819.009129:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201819.009130:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201819.009133:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1090 00000100:00080000:3.0:1788201819.009134:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201819.009136:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201819.009139:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201819.009145:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442810496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201819.009151:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201819.009153:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201819.009154:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201819.009156:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1090 00000100:00080000:3.0:1788201819.009158:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201819.009160:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201819.009163:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201819.009169:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442810624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201819.009205:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442810240:192.168.201.130@tcp:250:kworker.0 00000800:00000100:0.0:1788201819.009271:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201819.009314:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201819/real 1788201819] req@ffff90235268b100 x1875064442810240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201819.009335:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442810240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201819.009346:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201819.009350:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201819.009355:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442810240:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788201819.009363:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442810368:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788201819.009387:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201819.009401:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201819/real 1788201819] req@ffff902352688000 x1875064442810368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201819.009414:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442810368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201819.009423:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201819.009426:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201819.009430:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442810368:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788201819.009437:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442810496:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788201819.009459:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201819.009471:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201819/real 1788201819] req@ffff90235268a300 x1875064442810496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201819.009484:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442810496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201819.009493:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201819.009497:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201819.009500:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442810496:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788201819.009507:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442810624:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788201819.009527:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201819.009539:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201819/real 1788201819] req@ffff902352688e00 x1875064442810624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201819.009552:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442810624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201819.009561:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201819.009564:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201819.009568:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442810624:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201819.841089:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201820.865062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201821.889044:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201822.913066:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201823.937076:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201824.128965:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201824.128973:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201824.128977:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201824.128985:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1095 00000100:00080000:3.0:1788201824.128995:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201824.129000:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201824.129040:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442810752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201824.129064:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201824.129067:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201824.129068:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201824.129071:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1095 00000100:00080000:3.0:1788201824.129074:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201824.129076:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201824.129083:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201824.129092:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442810880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201824.129102:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201824.129105:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201824.129106:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201824.129108:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1095 00000100:00080000:3.0:1788201824.129111:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201824.129115:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201824.129118:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201824.129125:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442811008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201824.129132:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201824.129136:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788201824.129139:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442810752:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201824.129139:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201824.129147:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1095 00000100:00080000:3.0:1788201824.129154:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201824.129166:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201824.129170:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201824.129182:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442811136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788201824.129235:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201824.129281:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201824/real 1788201824] req@ffff90235268b800 x1875064442810752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201824.129305:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442810752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201824.129317:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201824.129323:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201824.129327:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442810752:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788201824.129337:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442810880:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788201824.129358:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201824.129368:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201824/real 1788201824] req@ffff902352689c00 x1875064442810880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201824.129377:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442810880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201824.129388:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201824.129391:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201824.129393:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442810880:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788201824.129417:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442811008:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788201824.129429:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201824.129436:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201824/real 1788201824] req@ffff90235268a680 x1875064442811008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201824.129444:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442811008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201824.129450:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201824.129452:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201824.129454:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442811008:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788201824.129458:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442811136:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788201824.129468:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201824.129474:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201824/real 1788201824] req@ffff902352689500 x1875064442811136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201824.129481:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442811136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201824.129486:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201824.129488:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201824.129491:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442811136:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201824.961024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201825.985050:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201827.009090:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201828.033062:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201829.057101:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201829.249043:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201829.249057:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201829.249062:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201829.249073:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1100 00000100:00080000:1.0:1788201829.249087:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201829.249093:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788201829.249140:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442811264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201829.249175:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201829.249180:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201829.249182:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201829.249186:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1100 00000100:00080000:1.0:1788201829.249189:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201829.249192:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201829.249203:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201829.249215:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442811392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201829.249224:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201829.249228:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201829.249229:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201829.249232:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1100 00000100:00080000:1.0:1788201829.249235:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201829.249237:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201829.249240:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201829.249249:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442811520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201829.249256:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201829.249260:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201829.249263:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788201829.249267:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442811264:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788201829.249268:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1100 00000100:00080000:1.0:1788201829.249271:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201829.249281:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201829.249287:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201829.249294:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442811648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788201829.249355:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201829.249412:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201829/real 1788201829] req@ffff90234519ea00 x1875064442811264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201829.249431:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442811264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201829.249441:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201829.249445:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201829.249449:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442811264:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788201829.249456:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442811392:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788201829.249475:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201829.249486:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201829/real 1788201829] req@ffff90234519fb80 x1875064442811392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201829.249496:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442811392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201829.249503:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201829.249506:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201829.249509:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442811392:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788201829.249532:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442811520:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788201829.249548:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201829.249557:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201829/real 1788201829] req@ffff90234519c700 x1875064442811520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201829.249567:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442811520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201829.249575:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201829.249578:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201829.249581:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442811520:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788201829.249586:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442811648:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788201829.249599:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201829.249607:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201829/real 1788201829] req@ffff90234519df80 x1875064442811648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201829.249617:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442811648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201829.249623:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201829.249626:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201829.249659:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442811648:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201830.081014:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201831.105109:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201832.129048:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201833.153071:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201834.177005:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201834.369040:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201834.369054:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201834.369060:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201834.369074:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1105 00000100:00080000:2.0:1788201834.369088:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201834.369096:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788201834.369141:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442811776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201834.369178:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201834.369181:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201834.369183:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201834.369189:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1105 00000100:00080000:2.0:1788201834.369194:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201834.369198:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788201834.369209:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201834.369228:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442811904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201834.369230:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442811776:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788201834.369257:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201834.369262:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201834.369265:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201834.369268:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1105 00000100:00080000:2.0:1788201834.369273:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201834.369276:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788201834.369280:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201834.369291:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442812032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201834.369298:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201834.369302:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201834.369304:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201834.369308:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1105 00000100:00080000:2.0:1788201834.369312:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201834.369315:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788201834.369319:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201834.369330:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442812160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201834.369342:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201834.369391:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201834/real 1788201834] req@ffff90234598c700 x1875064442811776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201834.369408:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442811776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201834.369416:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201834.369420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201834.369423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442811776:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201834.369448:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442811904:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201834.369475:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201834.369490:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201834/real 1788201834] req@ffff90234598dc00 x1875064442811904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201834.369512:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442811904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201834.369521:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201834.369525:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201834.369528:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442811904:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201834.369536:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442812032:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201834.369573:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201834.369587:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201834/real 1788201834] req@ffff90234598f800 x1875064442812032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201834.369600:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442812032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201834.369609:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201834.369612:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201834.369614:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442812032:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201834.369619:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442812160:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201834.369640:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201834.369650:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201834/real 1788201834] req@ffff90234598d880 x1875064442812160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201834.369662:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442812160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201834.369676:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201834.369681:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201834.369688:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442812160:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201835.201055:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201836.225045:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201837.249027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201838.273046:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201839.297067:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201839.489042:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201839.489052:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201839.489056:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201839.489067:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1111 00000100:00080000:3.0:1788201839.489079:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201839.489086:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201839.489133:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442812288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201839.489161:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201839.489166:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201839.489168:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201839.489171:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1111 00000100:00080000:3.0:1788201839.489173:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201839.489176:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201839.489192:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201839.489203:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442812416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201839.489211:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442812288:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201839.489214:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201839.489221:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201839.489222:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201839.489226:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1111 00000100:00080000:3.0:1788201839.489229:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201839.489231:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201839.489235:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201839.489245:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442812544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201839.489252:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201839.489255:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201839.489256:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201839.489258:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1111 00000100:00080000:3.0:1788201839.489261:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201839.489264:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201839.489268:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201839.489275:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442812672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201839.489338:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442812416:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201839.489368:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442812544:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201839.489388:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442812672:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201840.321054:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201841.345023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201842.368993:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788201842.561153:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:0.0:1788201842.561188:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788201842.561200:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201842.561234:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201842.561244:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201842.561253:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:1.0:1788201842.561278:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201839/real 1788201842] req@ffff902352689180 x1875064442812416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201842.561305:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442812416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201842.561318:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201842.561323:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201842.561328:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442812416:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201842.561337:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201839/real 1788201842] req@ffff902352689880 x1875064442812544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201842.561356:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442812544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201842.561364:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201842.561368:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201842.561372:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442812544:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201842.561379:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201839/real 1788201842] req@ffff902352688380 x1875064442812672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201842.561390:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442812672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201842.561423:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201842.561428:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201842.561431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442812672:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201842.561439:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201839/real 1788201842] req@ffff902352688700 x1875064442812288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201842.561453:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442812288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201842.561461:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201842.561465:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201842.561468:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442812288:192.168.201.130@tcp:250:kworker.0 00010000:00010000:2.0:1788201843.392987:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201844.417003:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201844.609049:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201844.609062:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201844.609065:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201844.609074:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1116 00000100:00080000:0.0:1788201844.609084:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201844.609091:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788201844.609128:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442812800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201844.609160:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201844.609165:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201844.609167:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201844.609171:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1116 00000100:00080000:0.0:1788201844.609176:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201844.609181:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201844.609190:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201844.609204:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442812928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201844.609218:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442812800:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788201844.609219:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201844.609230:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201844.609232:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201844.609237:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1116 00000100:00080000:0.0:1788201844.609241:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201844.609246:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201844.609253:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201844.609269:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442813056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201844.609281:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201844.609284:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201844.609286:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201844.609290:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1116 00000800:00000100:1.0:1788201844.609298:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788201844.609301:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201844.609309:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201844.609316:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201844.609330:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442813184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201844.609349:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201844/real 1788201844] req@ffff9023450b8700 x1875064442812800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201844.609388:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442812800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201844.609399:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201844.609404:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201844.609409:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442812800:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201844.609439:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442812928:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201844.609464:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201844.609478:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201844/real 1788201844] req@ffff9023450b9c00 x1875064442812928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201844.609491:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442812928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201844.609499:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201844.609503:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201844.609506:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442812928:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201844.609513:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442813056:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201844.609532:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201844.609542:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201844/real 1788201844] req@ffff9023450b9f80 x1875064442813056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201844.609561:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442813056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201844.609571:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201844.609575:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201844.609578:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442813056:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201844.609584:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442813184:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201844.609603:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201844.609614:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201844/real 1788201844] req@ffff9023450bad80 x1875064442813184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201844.609626:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442813184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201844.609634:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201844.609638:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201844.609642:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442813184:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201845.441038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201846.465059:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201847.489019:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201848.513051:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201849.537000:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201849.729026:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201849.729036:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201849.729040:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201849.729051:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1121 00000100:00080000:3.0:1788201849.729064:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201849.729071:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201849.729127:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442813312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201849.729160:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201849.729164:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201849.729166:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201849.729170:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1121 00000100:00080000:3.0:1788201849.729174:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201849.729178:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201849.729188:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201849.729201:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442813440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201849.729212:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442813312:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201849.729214:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201849.729221:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201849.729223:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201849.729228:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1121 00000100:00080000:3.0:1788201849.729233:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201849.729236:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201849.729242:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201849.729257:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442813568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201849.729267:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788201849.729286:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788201849.729288:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201849.729292:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201849.729299:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1121 00000100:00080000:3.0:1788201849.729307:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201849.729312:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201849.729318:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201849.729326:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201849/real 1788201849] req@ffff90235268bb80 x1875064442813312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201849.729335:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442813696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201849.729356:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442813312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201849.729365:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201849.729368:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201849.729372:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442813312:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201849.729392:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442813440:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201849.729409:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201849.729417:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201849/real 1788201849] req@ffff90235268ad80 x1875064442813440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201849.729425:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442813440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201849.729431:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201849.729433:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201849.729435:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442813440:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201849.729439:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442813568:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201849.729449:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201849.729455:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201849/real 1788201849] req@ffff90235268b480 x1875064442813568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201849.729462:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442813568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201849.729471:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201849.729474:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201849.729496:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442813568:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201849.729501:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442813696:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201849.729513:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201849.729521:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201849/real 1788201849] req@ffff902352688a80 x1875064442813696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201849.729530:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442813696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201849.729539:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201849.729543:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201849.729546:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442813696:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201850.561017:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201851.585056:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201852.609029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201853.633038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201854.657069:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201854.848974:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201854.848982:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201854.848985:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201854.848994:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1126 00000100:00080000:3.0:1788201854.849004:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201854.849011:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201854.849044:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064442813824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201854.849065:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201854.849070:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201854.849071:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201854.849074:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1126 00000100:00080000:3.0:1788201854.849076:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201854.849079:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201854.849085:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201854.849095:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442813952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201854.849101:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201854.849103:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201854.849104:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201854.849107:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1126 00000100:00080000:3.0:1788201854.849108:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201854.849110:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201854.849113:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201854.849129:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442814080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201854.849141:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788201854.849143:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442813824:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201854.849145:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201854.849150:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201854.849156:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1126 00000100:00080000:3.0:1788201854.849166:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201854.849171:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201854.849177:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201854.849214:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442814208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201854.849224:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201854.849258:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201854/real 1788201854] req@ffff90235268aa00 x1875064442813824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201854.849272:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064442813824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201854.849283:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201854.849288:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201854.849292:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442813824:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201854.849300:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442813952:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201854.849324:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201854.849336:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201854/real 1788201854] req@ffff902352688a80 x1875064442813952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201854.849346:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442813952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201854.849352:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201854.849354:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201854.849357:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442813952:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201854.849375:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442814080:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201854.849389:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201854.849395:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201854/real 1788201854] req@ffff90235268b480 x1875064442814080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201854.849403:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442814080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201854.849408:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201854.849410:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201854.849412:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442814080:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201854.849417:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442814208:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201854.849433:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201854.849444:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201854/real 1788201854] req@ffff90235268ad80 x1875064442814208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201854.849457:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442814208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201854.849466:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201854.849469:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201854.849472:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442814208:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201855.681012:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201856.705026:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201857.729041:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201858.753029:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201859.777053:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201859.968993:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201859.969001:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201859.969004:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201859.969013:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1131 00000100:00080000:3.0:1788201859.969023:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201859.969028:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201859.969060:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442814336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201859.969090:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201859.969122:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201859.969125:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201859.969130:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1131 00000100:00080000:3.0:1788201859.969135:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201859.969145:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201859.969154:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201859.969163:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442814336:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788201859.969168:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442814464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201859.969178:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201859.969181:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201859.969182:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201859.969185:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1131 00000100:00080000:3.0:1788201859.969189:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201859.969192:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201859.969195:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201859.969204:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442814592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201859.969211:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201859.969214:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201859.969215:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201859.969217:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1131 00000100:00080000:3.0:1788201859.969219:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201859.969221:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201859.969224:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788201859.969227:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788201859.969235:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442814720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201859.969265:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201859/real 1788201859] req@ffff90235268bb80 x1875064442814336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201859.969281:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442814336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201859.969288:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201859.969292:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201859.969295:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442814336:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201859.969313:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442814464:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201859.969327:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201859.969335:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201859/real 1788201859] req@ffff902352688700 x1875064442814464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201859.969344:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442814464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201859.969349:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201859.969351:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201859.969354:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442814464:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201859.969357:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442814592:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201859.969367:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201859.969372:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201859/real 1788201859] req@ffff902352688380 x1875064442814592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201859.969379:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442814592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201859.969385:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201859.969386:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201859.969388:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442814592:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201859.969392:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442814720:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201859.969401:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201859.969406:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201859/real 1788201859] req@ffff902352689880 x1875064442814720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201859.969413:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442814720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201859.969425:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201859.969428:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201859.969431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442814720:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201860.801029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201861.825055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201862.849031:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201863.873136:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201864.897025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201865.088967:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201865.088976:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201865.088986:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201865.088994:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1136 00000100:00080000:1.0:1788201865.089004:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201865.089031:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788201865.089065:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442814848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201865.089091:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201865.089095:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201865.089097:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201865.089102:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1136 00000100:00080000:1.0:1788201865.089107:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201865.089111:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201865.089121:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201865.089137:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442814976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201865.089147:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442814848:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788201865.089154:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201865.089157:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201865.089161:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201865.089165:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1136 00000100:00080000:1.0:1788201865.089168:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201865.089170:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201865.089175:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201865.089184:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442815104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201865.089191:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201865.089194:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201865.089195:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201865.089197:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1136 00000100:00080000:1.0:1788201865.089203:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201865.089205:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201865.089208:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788201865.089217:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201865.089218:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442815232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201865.089255:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201865/real 1788201865] req@ffff90234519ed80 x1875064442814848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201865.089269:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442814848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201865.089277:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201865.089280:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201865.089282:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442814848:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788201865.089300:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442814976:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788201865.089315:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201865.089324:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201865/real 1788201865] req@ffff90234519d880 x1875064442814976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201865.089332:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442814976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201865.089337:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201865.089340:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201865.089342:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442814976:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788201865.089345:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442815104:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788201865.089356:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201865.089362:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201865/real 1788201865] req@ffff90234519c000 x1875064442815104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201865.089369:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442815104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201865.089374:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201865.089376:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201865.089378:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442815104:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788201865.089381:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442815232:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788201865.089390:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788201865.089396:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201865/real 1788201865] req@ffff90234519e300 x1875064442815232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201865.089407:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442815232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201865.089412:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201865.089414:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788201865.089416:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442815232:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201865.921047:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201866.945042:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201867.969086:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201868.993028:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201870.017121:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201870.208984:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201870.208994:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201870.208997:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201870.209007:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1141 00000100:00080000:3.0:1788201870.209016:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201870.209022:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201870.209059:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442815360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201870.209085:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201870.209090:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201870.209092:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201870.209097:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1141 00000100:00080000:3.0:1788201870.209101:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201870.209105:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201870.209117:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201870.209126:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442815488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201870.209132:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442815360:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201870.209138:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201870.209144:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201870.209145:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201870.209148:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1141 00000100:00080000:3.0:1788201870.209150:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201870.209152:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201870.209156:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201870.209165:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442815616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201870.209171:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201870.209174:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201870.209175:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201870.209178:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1141 00000100:00080000:3.0:1788201870.209181:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201870.209183:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201870.209187:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201870.209195:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442815744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201870.209202:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201870.209241:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201870/real 1788201870] req@ffff902352689180 x1875064442815360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201870.209256:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442815360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201870.209265:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201870.209268:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201870.209272:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442815360:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201870.209293:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442815488:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201870.209313:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201870.209322:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201870/real 1788201870] req@ffff902352689500 x1875064442815488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201870.209332:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442815488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201870.209339:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201870.209341:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201870.209343:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442815488:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201870.209349:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442815616:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201870.209364:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201870.209374:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201870/real 1788201870] req@ffff90235268a680 x1875064442815616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201870.209388:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442815616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201870.209395:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201870.209397:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201870.209401:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442815616:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201870.209405:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442815744:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201870.209417:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201870.209425:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201870/real 1788201870] req@ffff902352689c00 x1875064442815744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201870.209434:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442815744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201870.209439:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201870.209442:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201870.209444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442815744:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201871.041031:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201872.065058:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201873.089035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201874.113034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201875.137053:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201875.328993:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201875.329002:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201875.329005:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201875.329023:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1146 00000100:00080000:0.0:1788201875.329032:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201875.329038:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788201875.329078:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442815872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201875.329110:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201875.329119:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201875.329121:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201875.329124:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1146 00000100:00080000:0.0:1788201875.329128:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201875.329131:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201875.329139:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201875.329152:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442816000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201875.329157:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442815872:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788201875.329179:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201875.329183:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201875.329185:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201875.329191:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1146 00000100:00080000:0.0:1788201875.329195:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201875.329201:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201875.329206:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201875.329217:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442816128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201875.329224:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201875.329230:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1788201875.329230:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788201875.329233:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201875.329238:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1146 00000100:00080000:0.0:1788201875.329241:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201875.329244:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201875.329247:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201875.329255:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442816256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201875.329281:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201875/real 1788201875] req@ffff9023450b9180 x1875064442815872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201875.329300:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442815872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201875.329308:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201875.329312:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201875.329315:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442815872:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201875.329334:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442816000:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201875.329352:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201875.329361:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201875/real 1788201875] req@ffff9023450b8000 x1875064442816000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201875.329370:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442816000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201875.329375:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201875.329377:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201875.329380:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442816000:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201875.329383:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442816128:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201875.329395:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201875.329401:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201875/real 1788201875] req@ffff9023450ba300 x1875064442816128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201875.329408:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442816128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201875.329414:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201875.329416:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201875.329418:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442816128:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201875.329422:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442816256:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201875.329432:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201875.329437:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201875/real 1788201875] req@ffff9023450baa00 x1875064442816256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201875.329445:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442816256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201875.329450:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201875.329451:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201875.329453:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442816256:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201876.161047:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201877.185106:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201878.209062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201879.233063:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201880.257040:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201880.448979:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201880.448988:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201880.448992:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201880.449001:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1151 00000100:00080000:3.0:1788201880.449009:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201880.449015:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201880.449055:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442816384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201880.449084:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201880.449087:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201880.449089:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201880.449094:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1151 00000100:00080000:3.0:1788201880.449098:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201880.449102:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201880.449112:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201880.449126:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442816384:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788201880.449131:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442816512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201880.449144:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201880.449147:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201880.449149:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201880.449152:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1151 00000100:00080000:3.0:1788201880.449156:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201880.449159:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201880.449164:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201880.449173:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442816640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201880.449180:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201880.449183:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201880.449184:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788201880.449188:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788201880.449189:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1151 00000100:00080000:3.0:1788201880.449194:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201880.449198:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201880.449202:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201880.449210:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442816768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201880.449230:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201880/real 1788201880] req@ffff90235268b800 x1875064442816384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201880.449246:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442816384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201880.449254:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201880.449256:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201880.449259:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442816384:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201880.449277:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442816512:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201880.449293:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201880.449301:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201880/real 1788201880] req@ffff902352688e00 x1875064442816512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201880.449309:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442816512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201880.449315:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201880.449317:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201880.449319:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442816512:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201880.449332:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442816640:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201880.449344:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201880.449350:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201880/real 1788201880] req@ffff90235268a300 x1875064442816640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201880.449358:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442816640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201880.449363:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201880.449365:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201880.449367:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442816640:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201880.449371:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442816768:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201880.449380:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201880.449386:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201880/real 1788201880] req@ffff902352688000 x1875064442816768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201880.449393:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442816768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201880.449398:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201880.449401:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201880.449403:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442816768:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201881.281042:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201882.305079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201883.329094:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201884.353020:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201885.377071:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201885.569001:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201885.569010:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201885.569014:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201885.569024:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1157 00000100:00080000:3.0:1788201885.569034:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201885.569041:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201885.569075:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442816896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201885.569099:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201885.569104:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201885.569105:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201885.569108:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1157 00000100:00080000:3.0:1788201885.569110:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201885.569113:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201885.569120:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201885.569131:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442817024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201885.569140:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201885.569143:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201885.569144:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201885.569148:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1157 00000100:00080000:3.0:1788201885.569151:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788201885.569153:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442816896:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201885.569154:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201885.569161:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201885.569169:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442817152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201885.569176:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201885.569179:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201885.569180:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201885.569183:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1157 00000100:00080000:3.0:1788201885.569184:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201885.569187:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201885.569191:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201885.569199:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442817280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201885.569223:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201885.569265:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201885/real 1788201885] req@ffff90235268b100 x1875064442816896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201885.569280:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442816896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201885.569288:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201885.569291:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201885.569294:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442816896:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201885.569313:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442817024:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201885.569328:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201885.569337:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201885/real 1788201885] req@ffff902352688000 x1875064442817024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201885.569345:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442817024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201885.569355:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201885.569357:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201885.569360:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442817024:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201885.569363:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442817152:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201885.569374:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201885.569380:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201885/real 1788201885] req@ffff90235268a300 x1875064442817152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201885.569387:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442817152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201885.569393:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201885.569394:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201885.569396:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442817152:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201885.569400:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442817280:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201885.569409:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201885.569415:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201885/real 1788201885] req@ffff902352688e00 x1875064442817280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201885.569422:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442817280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201885.569427:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201885.569428:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201885.569431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442817280:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201886.401046:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201887.425015:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201888.449024:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201889.473070:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201890.497023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201890.688949:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201890.688961:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201890.688965:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201890.688976:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1162 00000100:00080000:3.0:1788201890.688988:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201890.688995:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201890.689038:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442817408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201890.689067:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201890.689070:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201890.689072:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201890.689076:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1162 00000100:00080000:3.0:1788201890.689078:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201890.689087:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201890.689098:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201890.689109:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442817536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201890.689118:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788201890.689124:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442817408:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201890.689124:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201890.689127:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201890.689135:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1162 00000100:00080000:3.0:1788201890.689139:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201890.689142:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201890.689151:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201890.689163:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442817664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201890.689171:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201890.689174:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201890.689175:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201890.689178:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1162 00000100:00080000:3.0:1788201890.689180:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201890.689182:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201890.689186:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201890.689194:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442817792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201890.689204:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201890.689253:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201890/real 1788201890] req@ffff90235268b800 x1875064442817408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201890.689272:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442817408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201890.689281:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201890.689285:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201890.689288:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442817408:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201890.689317:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442817536:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201890.689334:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201890.689343:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201890/real 1788201890] req@ffff902352689c00 x1875064442817536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201890.689352:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442817536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201890.689358:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201890.689361:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201890.689363:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442817536:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201890.689369:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442817664:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201890.689381:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201890.689387:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201890/real 1788201890] req@ffff90235268a680 x1875064442817664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201890.689395:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442817664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201890.689401:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201890.689402:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201890.689404:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442817664:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201890.689408:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442817792:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201890.689418:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201890.689424:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201890/real 1788201890] req@ffff902352689500 x1875064442817792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201890.689431:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442817792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201890.689444:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201890.689446:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201890.689448:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442817792:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201891.521055:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201892.545022:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201893.569032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201894.593034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201895.617054:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201895.809119:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201895.809130:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201895.809135:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201895.809146:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1167 00000100:00080000:3.0:1788201895.809158:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201895.809165:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201895.809203:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442817920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201895.809237:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201895.809248:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201895.809250:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201895.809255:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1167 00000100:00080000:3.0:1788201895.809259:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201895.809264:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201895.809275:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201895.809282:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442817920:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788201895.809293:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442818048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201895.809306:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201895.809311:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201895.809313:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201895.809318:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1167 00000100:00080000:3.0:1788201895.809322:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201895.809326:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201895.809332:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788201895.809341:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788201895.809347:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442818176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201895.809364:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201895.809368:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201895.809371:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201895.809375:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1167 00000100:00080000:3.0:1788201895.809380:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788201895.809386:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201895/real 1788201895] req@ffff902352689180 x1875064442817920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201895.809387:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201895.809397:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201895.809408:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442817920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201895.809415:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442818304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201895.809421:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201895.809429:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201895.809434:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442817920:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201895.809455:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442818048:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201895.809497:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201895.809508:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201895/real 1788201895] req@ffff902352689880 x1875064442818048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201895.809517:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442818048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201895.809523:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201895.809525:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201895.809528:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442818048:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201895.809532:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442818176:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201895.809546:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201895.809552:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201895/real 1788201895] req@ffff902352688380 x1875064442818176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201895.809564:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442818176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201895.809569:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201895.809571:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201895.809573:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442818176:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201895.809577:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442818304:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201895.809593:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201895.809604:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201895/real 1788201895] req@ffff902352688700 x1875064442818304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201895.809617:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442818304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201895.809626:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201895.809629:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201895.809633:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442818304:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201896.641066:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201897.665091:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201898.688992:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201899.713020:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201900.737054:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201900.929006:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201900.929014:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201900.929018:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201900.929026:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1172 00000100:00080000:2.0:1788201900.929036:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201900.929041:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788201900.929074:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064442818432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201900.929098:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201900.929101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201900.929102:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201900.929105:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1172 00000100:00080000:2.0:1788201900.929107:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201900.929109:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788201900.929117:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201900.929125:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064442818560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201900.929132:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201900.929135:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201900.929136:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201900.929139:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1172 00000100:00080000:2.0:1788201900.929141:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201900.929144:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788201900.929147:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201900.929154:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442818432:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788201900.929158:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442818688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201900.929167:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201900.929170:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201900.929171:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201900.929173:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1172 00000100:00080000:2.0:1788201900.929176:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201900.929179:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788201900.929182:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201900.929194:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442818816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201900.929223:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201900.929263:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201900/real 1788201900] req@ffff90234598ca80 x1875064442818432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201900.929279:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064442818432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201900.929290:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201900.929295:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201900.929299:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442818432:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201900.929307:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442818560:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201900.929331:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201900.929347:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201900/real 1788201900] req@ffff90234598e680 x1875064442818560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201900.929357:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064442818560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201900.929362:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201900.929365:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201900.929367:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442818560:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201900.929403:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442818688:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201900.929419:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201900.929427:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201900/real 1788201900] req@ffff90234598d880 x1875064442818688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201900.929435:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442818688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201900.929441:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201900.929443:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201900.929446:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442818688:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201900.929450:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442818816:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201900.929460:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201900.929466:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201900/real 1788201900] req@ffff90234598f800 x1875064442818816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201900.929473:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442818816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201900.929478:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201900.929480:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201900.929482:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442818816:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201901.761083:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201902.785069:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201903.809042:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201904.833056:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201905.857069:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201906.049032:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201906.049043:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201906.049048:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201906.049060:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1177 00000100:00080000:0.0:1788201906.049075:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201906.049086:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788201906.049137:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442818944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201906.049163:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201906.049166:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201906.049168:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201906.049171:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1177 00000100:00080000:0.0:1788201906.049175:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201906.049179:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201906.049189:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201906.049202:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442819072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201906.049210:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201906.049212:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201906.049214:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201906.049217:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1177 00000100:00080000:0.0:1788201906.049219:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201906.049221:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201906.049224:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201906.049235:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442819200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201906.049242:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201906.049245:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201906.049247:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201906.049251:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1177 00000100:00080000:0.0:1788201906.049253:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201906.049256:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201906.049260:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201906.049263:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442818944:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788201906.049271:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442819328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201906.049618:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442819072:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788201906.049656:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442819200:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788201906.050573:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442819328:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201906.881021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201907.905021:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201908.929006:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788201909.121320:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788201909.121385:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788201909.121408:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788201909.121460:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788201909.121471:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788201909.121480:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788201909.121539:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201906/real 1788201909] req@ffff9023450b9180 x1875064442819328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201909.121580:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442819328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201909.121596:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201909.121602:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788201909.121607:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442819328:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788201909.121618:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201906/real 1788201909] req@ffff9023450baa00 x1875064442818944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201909.121634:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442818944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201909.121642:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201909.121646:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788201909.121651:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442818944:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788201909.121664:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201906/real 1788201909] req@ffff9023450ba300 x1875064442819072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201909.121678:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442819072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201909.121687:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201909.121692:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788201909.121696:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442819072:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788201909.121703:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201906/real 1788201909] req@ffff9023450b8000 x1875064442819200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201909.121717:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442819200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201909.121725:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201909.121730:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788201909.121734:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442819200:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201909.952994:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201910.977040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201911.169013:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201911.169027:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201911.169031:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201911.169044:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1182 00000100:00080000:3.0:1788201911.169059:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201911.169069:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201911.169121:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442819456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201911.169160:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201911.169164:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201911.169166:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201911.169171:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1182 00000100:00080000:3.0:1788201911.169175:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201911.169179:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201911.169198:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201911.169225:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442819584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201911.169237:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788201911.169241:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442819456:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201911.169242:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201911.169247:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201911.169254:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1182 00000100:00080000:3.0:1788201911.169258:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201911.169262:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201911.169268:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201911.169281:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442819712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201911.169292:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201911.169295:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201911.169297:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201911.169301:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1182 00000100:00080000:3.0:1788201911.169307:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201911.169311:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201911.169316:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201911.169326:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442819840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201911.169354:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201911.169417:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201911/real 1788201911] req@ffff90235268bb80 x1875064442819456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201911.169438:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442819456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201911.169452:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201911.169458:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201911.169462:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442819456:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201911.169497:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442819584:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201911.169525:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201911.169539:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201911/real 1788201911] req@ffff90235268ad80 x1875064442819584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201911.169552:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442819584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201911.169560:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201911.169563:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201911.169567:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442819584:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201911.169574:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442819712:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201911.169594:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201911.169606:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201911/real 1788201911] req@ffff90235268b480 x1875064442819712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201911.169617:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442819712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201911.169625:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201911.169628:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201911.169631:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442819712:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201911.169636:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442819840:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201911.169672:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201911.169683:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201911/real 1788201911] req@ffff902352688a80 x1875064442819840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201911.169694:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442819840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201911.169703:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201911.169706:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201911.169709:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442819840:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201912.000998:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201913.025066:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201914.049092:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201915.073111:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201916.097152:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201916.289010:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201916.289022:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201916.289026:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201916.289049:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1187 00000100:00080000:3.0:1788201916.289062:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201916.289070:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201916.289132:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064442819968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201916.289175:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201916.289183:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201916.289185:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201916.289191:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1187 00000100:00080000:3.0:1788201916.289196:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201916.289217:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201916.289230:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201916.289237:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442819968:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788201916.289251:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442820096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201916.289262:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201916.289284:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201916.289286:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201916.289289:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1187 00000100:00080000:3.0:1788201916.289293:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201916.289296:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201916.289308:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201916.289319:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442820224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201916.289327:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201916.289330:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201916.289332:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788201916.289337:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788201916.289339:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1187 00000100:00080000:3.0:1788201916.289344:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201916.289347:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201916.289351:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201916.289360:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442820352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201916.289387:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201916/real 1788201916] req@ffff90235268aa00 x1875064442819968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201916.289405:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064442819968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201916.289414:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201916.289418:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201916.289421:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442819968:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201916.289455:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442820096:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201916.289475:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201916.289486:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201916/real 1788201916] req@ffff902352688a80 x1875064442820096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201916.289495:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442820096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201916.289518:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201916.289520:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201916.289523:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442820096:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201916.289527:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442820224:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201916.289548:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201916.289560:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201916/real 1788201916] req@ffff90235268b480 x1875064442820224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201916.289590:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442820224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201916.289600:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201916.289603:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201916.289607:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442820224:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201916.289612:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442820352:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201916.289639:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201916.289648:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201916/real 1788201916] req@ffff90235268ad80 x1875064442820352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201916.289662:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442820352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201916.289668:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201916.289671:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201916.289673:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442820352:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201917.121047:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201918.145160:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201919.169116:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201920.193119:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201921.217070:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201921.408979:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201921.408987:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201921.408991:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201921.409000:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1192 00000100:00080000:0.0:1788201921.409008:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201921.409014:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788201921.409070:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442820480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201921.409093:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201921.409096:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201921.409097:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201921.409100:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1192 00000100:00080000:0.0:1788201921.409103:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201921.409106:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201921.409113:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201921.409122:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442820608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201921.409140:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201921.409143:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201921.409144:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201921.409147:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1192 00000100:00080000:0.0:1788201921.409149:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201921.409152:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201921.409155:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201921.409162:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442820736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201921.409175:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788201921.409175:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442820480:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788201921.409193:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201921.409196:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201921.409199:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1192 00000100:00080000:0.0:1788201921.409202:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201921.409204:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201921.409207:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201921.409214:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442820864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201921.409277:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201921.409332:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201921/real 1788201921] req@ffff9023450bad80 x1875064442820480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201921.409351:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442820480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201921.409363:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201921.409368:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201921.409373:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442820480:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201921.409390:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442820608:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201921.409416:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201921.409428:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201921/real 1788201921] req@ffff9023450b9f80 x1875064442820608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201921.409436:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442820608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201921.409447:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201921.409450:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201921.409452:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442820608:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201921.409482:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442820736:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201921.409504:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201921.409516:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201921/real 1788201921] req@ffff9023450b9c00 x1875064442820736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201921.409527:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442820736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201921.409533:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201921.409536:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201921.409538:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442820736:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201921.409543:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442820864:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201921.409559:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201921.409570:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201921/real 1788201921] req@ffff9023450b8700 x1875064442820864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201921.409582:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442820864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201921.409599:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201921.409603:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201921.409607:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442820864:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201922.241051:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201923.265079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201924.289056:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201925.312994:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201926.337060:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201926.529015:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201926.529024:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201926.529035:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201926.529044:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1198 00000100:00080000:2.0:1788201926.529053:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201926.529060:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788201926.529095:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442820992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201926.529158:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201926.529166:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201926.529169:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201926.529177:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1198 00000100:00080000:2.0:1788201926.529216:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201926.529227:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788201926.529247:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201926.529283:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442820992:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788201926.529286:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442821120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201926.529303:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201926.529307:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201926.529310:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201926.529315:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1198 00000100:00080000:2.0:1788201926.529321:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201926.529327:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788201926.529333:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788201926.529349:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201926.529351:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442821248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201926.529361:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201926.529364:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201926.529367:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201926.529370:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1198 00000100:00080000:2.0:1788201926.529374:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201926.529377:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788201926.529392:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201926.529403:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201926/real 1788201926] req@ffff90234598dc00 x1875064442820992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201926.529404:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442821376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201926.529461:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442820992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201926.529471:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201926.529475:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201926.529479:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442820992:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201926.529499:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442821120:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201926.529520:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201926.529553:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201926/real 1788201926] req@ffff90234598c700 x1875064442821120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201926.529589:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442821120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201926.529604:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201926.529610:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201926.529617:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442821120:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201926.529629:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442821248:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201926.529672:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201926.529705:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201926/real 1788201926] req@ffff90234598ce00 x1875064442821248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201926.529714:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442821248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201926.529722:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201926.529724:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201926.529727:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442821248:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201926.529731:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442821376:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201926.529756:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201926.529763:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201926/real 1788201926] req@ffff90234598e300 x1875064442821376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201926.529772:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442821376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201926.529777:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201926.529779:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201926.529781:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442821376:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201927.361148:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201928.385030:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201929.409058:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201930.433154:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201931.457038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201931.649081:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201931.649095:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201931.649101:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201931.649114:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1203 00000100:00080000:2.0:1788201931.649128:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201931.649137:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788201931.649194:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442821504/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201931.649230:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201931.649234:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201931.649237:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201931.649243:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1203 00000100:00080000:2.0:1788201931.649247:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201931.649252:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788201931.649264:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201931.649282:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442821632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201931.649307:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201931.649312:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201931.649316:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201931.649322:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1203 00000100:00080000:2.0:1788201931.649326:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201931.649334:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788201931.649342:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442821504:192.168.201.130@tcp:250:kworker.0 00000008:00100000:2.0:1788201931.649346:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201931.649374:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d500 x1875064442821760/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201931.649391:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201931.649396:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201931.649398:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201931.649404:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1203 00000100:00080000:2.0:1788201931.649410:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201931.649415:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788201931.649424:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201931.649443:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442821888/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788201931.649473:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201931.649567:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201931/real 1788201931] req@ffff90234598c380 x1875064442821504/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201931.649591:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442821504/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201931.649607:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201931.649612:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201931.649618:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442821504:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201931.649629:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442821632:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788201931.649652:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201931.649664:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201931/real 1788201931] req@ffff90234598d180 x1875064442821632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201931.649676:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442821632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201931.649684:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201931.649687:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201931.649690:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442821632:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788201931.649722:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442821760:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201931.649738:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201931.649747:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201931/real 1788201931] req@ffff90234598d500 x1875064442821760/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201931.649758:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d500 x1875064442821760/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201931.649772:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201931.649775:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201931.649778:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442821760:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788201931.649783:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442821888:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788201931.649796:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201931.649804:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201931/real 1788201931] req@ffff90234598f480 x1875064442821888/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201931.649814:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442821888/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201931.649822:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201931.649824:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788201931.649827:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442821888:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201932.481039:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201933.505049:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201934.529076:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201935.553053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201936.577067:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201936.768981:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201936.768990:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201936.768994:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201936.769002:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1208 00000100:00080000:1.0:1788201936.769011:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201936.769016:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788201936.769051:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442822016/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201936.769073:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201936.769076:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201936.769077:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201936.769080:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1208 00000100:00080000:1.0:1788201936.769082:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201936.769085:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201936.769092:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201936.769102:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442822144/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201936.769110:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201936.769113:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201936.769115:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788201936.769117:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442822016:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788201936.769119:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1208 00000100:00080000:1.0:1788201936.769123:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201936.769128:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201936.769130:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201936.769138:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442822272/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201936.769144:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201936.769147:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201936.769149:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201936.769151:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1208 00000100:00080000:1.0:1788201936.769157:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201936.769161:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201936.769166:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201936.769179:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442822400/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788201936.769180:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788201936.769219:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201936/real 1788201936] req@ffff90234519f800 x1875064442822016/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201936.769233:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442822016/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201936.769246:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201936.769251:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788201936.769255:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442822016:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788201936.769278:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442822144:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788201936.769301:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788201936.769311:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201936/real 1788201936] req@ffff90234519e300 x1875064442822144/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201936.769330:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442822144/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201936.769336:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201936.769338:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788201936.769341:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442822144:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788201936.769345:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442822272:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788201936.769358:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788201936.769365:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201936/real 1788201936] req@ffff90234519c000 x1875064442822272/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201936.769372:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442822272/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201936.769377:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201936.769379:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788201936.769381:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442822272:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788201936.769385:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442822400:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788201936.769394:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788201936.769400:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201936/real 1788201936] req@ffff90234519d880 x1875064442822400/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201936.769407:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442822400/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201936.769413:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201936.769414:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788201936.769416:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442822400:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201937.601014:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201938.625020:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201939.649104:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201940.673061:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201941.697005:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201941.888978:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201941.888988:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201941.888991:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201941.889000:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1213 00000100:00080000:1.0:1788201941.889009:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201941.889015:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788201941.889046:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442822528/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201941.889074:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201941.889078:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201941.889080:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201941.889083:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1213 00000100:00080000:1.0:1788201941.889086:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201941.889090:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201941.889098:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201941.889112:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442822656/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201941.889112:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442822528:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788201941.889131:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201941.889134:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201941.889135:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201941.889139:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1213 00000100:00080000:1.0:1788201941.889142:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201941.889144:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201941.889149:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201941.889159:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442822784/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201941.889166:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201941.889169:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201941.889170:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201941.889174:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1213 00000800:00000100:2.0:1788201941.889179:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788201941.889181:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201941.889186:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201941.889190:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201941.889199:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442822912/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201941.889220:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201941/real 1788201941] req@ffff90234519ed80 x1875064442822528/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201941.889256:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442822528/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201941.889268:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201941.889273:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201941.889279:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442822528:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788201941.889320:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442822656:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788201941.889340:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201941.889371:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201941/real 1788201941] req@ffff90234519df80 x1875064442822656/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201941.889386:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442822656/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201941.889395:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201941.889399:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201941.889402:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442822656:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201941.889410:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442822784:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201941.889433:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201941.889445:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201941/real 1788201941] req@ffff90234519c700 x1875064442822784/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201941.889459:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442822784/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201941.889469:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201941.889472:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201941.889476:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442822784:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788201941.889483:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442822912:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201941.889507:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201941.889518:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201941/real 1788201941] req@ffff90234519fb80 x1875064442822912/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201941.889527:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442822912/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201941.889533:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201941.889536:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201941.889539:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442822912:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201942.721017:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201943.745112:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201944.769060:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201945.793056:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201946.817040:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201947.009059:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201947.009070:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201947.009075:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201947.009085:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1218 00000100:00080000:0.0:1788201947.009098:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201947.009105:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788201947.009152:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442823040/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201947.009182:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201947.009185:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201947.009187:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201947.009190:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1218 00000100:00080000:0.0:1788201947.009192:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201947.009196:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201947.009205:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201947.009215:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442823168/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201947.009241:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201947.009247:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788201947.009255:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442823040:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788201947.009256:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201947.009276:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1218 00000100:00080000:0.0:1788201947.009282:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201947.009288:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201947.009297:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201947.009317:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442823296/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201947.009329:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201947.009336:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201947.009339:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788201947.009345:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788201947.009350:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1218 00000100:00080000:0.0:1788201947.009358:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201947.009381:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201947.009388:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201947.009410:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201947/real 1788201947] req@ffff9023450b8380 x1875064442823040/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201947.009420:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442823424/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201947.009432:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442823040/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201947.009452:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201947.009457:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201947.009460:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442823040:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788201947.009486:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442823168:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788201947.009503:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201947.009512:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201947/real 1788201947] req@ffff9023450bb800 x1875064442823168/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201947.009521:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442823168/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201947.009530:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201947.009533:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201947.009536:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442823168:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201947.009543:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442823296:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201947.009564:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201947.009577:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201947/real 1788201947] req@ffff9023450b8a80 x1875064442823296/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201947.009613:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442823296/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201947.009620:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201947.009622:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201947.009624:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442823296:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788201947.009629:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442823424:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201947.009663:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201947.009671:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201947/real 1788201947] req@ffff9023450b9500 x1875064442823424/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201947.009679:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442823424/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201947.009684:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201947.009687:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201947.009689:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442823424:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201947.841036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201948.865034:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201949.889078:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201950.912999:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201951.937039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201952.129022:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201952.129034:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201952.129039:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201952.129052:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1223 00000100:00080000:1.0:1788201952.129064:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201952.129072:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788201952.129119:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442823552/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201952.129150:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201952.129153:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201952.129155:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201952.129159:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1223 00000100:00080000:1.0:1788201952.129162:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201952.129166:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201952.129178:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201952.129189:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442823680/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201952.129197:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201952.129201:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201952.129202:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201952.129207:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1223 00000100:00080000:1.0:1788201952.129215:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788201952.129215:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442823552:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788201952.129223:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201952.129236:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201952.129252:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442823808/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201952.129261:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201952.129265:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201952.129267:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201952.129270:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1223 00000100:00080000:1.0:1788201952.129273:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201952.129278:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201952.129281:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201952.129291:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442823936/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201952.129300:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201952.129354:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201952/real 1788201952] req@ffff90234519ea00 x1875064442823552/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201952.129380:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442823552/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201952.129393:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201952.129399:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201952.129404:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442823552:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788201952.129431:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442823680:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788201952.129453:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201952.129463:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201952/real 1788201952] req@ffff90234519c380 x1875064442823680/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201952.129474:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442823680/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201952.129483:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201952.129486:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201952.129490:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442823680:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201952.129547:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442823808:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201952.129572:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201952.129591:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201952/real 1788201952] req@ffff90234519f480 x1875064442823808/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201952.129602:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442823808/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201952.129609:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201952.129612:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201952.129614:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442823808:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788201952.129620:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442823936:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201952.129633:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201952.129641:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201952/real 1788201952] req@ffff90234519dc00 x1875064442823936/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201952.129654:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442823936/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201952.129659:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201952.129661:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201952.129663:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442823936:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201952.961094:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201953.985063:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201955.009096:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201956.033073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201957.057044:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201957.249058:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201957.249071:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201957.249076:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201957.249087:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1228 00000100:00080000:3.0:1788201957.249099:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201957.249107:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201957.249153:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442824064/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201957.249184:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201957.249187:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201957.249188:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201957.249192:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1228 00000100:00080000:3.0:1788201957.249194:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201957.249198:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201957.249208:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201957.249217:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442824192/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201957.249224:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201957.249228:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201957.249229:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201957.249232:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1228 00000100:00080000:3.0:1788201957.249234:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201957.249236:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201957.249239:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201957.249247:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442824320/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201957.249255:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442824064:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201957.249257:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201957.249264:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201957.249266:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201957.249269:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1228 00000100:00080000:3.0:1788201957.249270:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201957.249273:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201957.249276:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201957.249284:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442824448/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201957.249334:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201957.249376:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201957/real 1788201957] req@ffff90235268bb80 x1875064442824064/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201957.249392:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442824064/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201957.249401:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201957.249427:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201957.249431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442824064:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788201957.249437:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442824192:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788201957.249456:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201957.249465:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201957/real 1788201957] req@ffff902352688700 x1875064442824192/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201957.249474:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442824192/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201957.249480:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201957.249482:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201957.249485:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442824192:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201957.249508:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442824320:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201957.249520:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201957.249526:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201957/real 1788201957] req@ffff902352688380 x1875064442824320/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201957.249534:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442824320/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201957.249540:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201957.249541:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201957.249543:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442824320:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788201957.249546:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442824448:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201957.249556:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201957.249562:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201957/real 1788201957] req@ffff902352689880 x1875064442824448/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201957.249569:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442824448/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201957.249574:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201957.249576:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201957.249578:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442824448:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201958.081120:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201959.105109:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201960.129043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201961.153096:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201962.177057:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201962.368995:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201962.369004:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201962.369007:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201962.369016:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1233 00000100:00080000:1.0:1788201962.369025:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201962.369031:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788201962.369065:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442824576/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201962.369097:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201962.369102:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201962.369103:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201962.369107:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1233 00000100:00080000:1.0:1788201962.369110:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201962.369113:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201962.369121:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201962.369134:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442824704/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201962.369144:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788201962.369145:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442824576:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788201962.369147:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201962.369151:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201962.369156:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1233 00000100:00080000:1.0:1788201962.369158:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201962.369161:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201962.369164:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201962.369173:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442824832/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201962.369180:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201962.369183:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201962.369184:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201962.369186:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1233 00000100:00080000:1.0:1788201962.369189:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201962.369191:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201962.369194:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201962.369202:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442824960/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201962.369209:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201962.369277:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201962/real 1788201962] req@ffff90234519f100 x1875064442824576/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201962.369295:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442824576/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201962.369303:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201962.369307:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201962.369310:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442824576:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788201962.369375:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442824704:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788201962.369408:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201962.369419:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201962/real 1788201962] req@ffff90234519dc00 x1875064442824704/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201962.369429:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442824704/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201962.369435:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201962.369437:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201962.369439:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442824704:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201962.369445:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442824832:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201962.369457:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201962.369464:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201962/real 1788201962] req@ffff90234519f480 x1875064442824832/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201962.369471:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442824832/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201962.369477:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201962.369478:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201962.369480:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442824832:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788201962.369484:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442824960:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201962.369493:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201962.369499:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201962/real 1788201962] req@ffff90234519c380 x1875064442824960/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201962.369507:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442824960/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201962.369512:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201962.369513:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201962.369515:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442824960:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201963.201067:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201964.225054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201965.249049:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201966.273026:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201967.297077:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201967.488994:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201967.489006:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201967.489010:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201967.489020:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1239 00000100:00080000:0.0:1788201967.489031:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201967.489038:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788201967.489075:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442825088/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201967.489104:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201967.489108:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201967.489110:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201967.489114:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1239 00000100:00080000:0.0:1788201967.489119:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201967.489123:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201967.489133:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201967.489145:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442825088:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788201967.489150:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442825216/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201967.489167:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201967.489171:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201967.489172:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201967.489177:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1239 00000100:00080000:0.0:1788201967.489182:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201967.489187:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788201967.489193:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201967.489206:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442825344/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201967.489217:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788201967.489225:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201967.489245:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201967.489254:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201967.489260:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1239 00000100:00080000:0.0:1788201967.489265:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201967.489270:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788201967.489275:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201967/real 1788201967] req@ffff9023450ba680 x1875064442825088/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788201967.489277:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201967.489296:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442825472/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201967.489298:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442825088/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201967.489323:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201967.489328:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201967.489332:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442825088:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788201967.489350:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442825216:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788201967.489373:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201967.489388:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201967/real 1788201967] req@ffff9023450b8e00 x1875064442825216/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201967.489401:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442825216/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201967.489408:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201967.489423:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201967.489430:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442825216:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201967.489436:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442825344:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201967.489458:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201967.489471:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201967/real 1788201967] req@ffff9023450bbb80 x1875064442825344/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201967.489497:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442825344/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201967.489503:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201967.489505:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201967.489507:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442825344:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788201967.489512:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442825472:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201967.489527:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201967.489535:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201967/real 1788201967] req@ffff9023450bb480 x1875064442825472/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201967.489542:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442825472/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201967.489548:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201967.489550:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201967.489552:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442825472:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201968.321022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201969.345097:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201970.369034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201971.393048:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201972.417097:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201972.609046:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201972.609060:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201972.609065:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201972.609078:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1244 00000100:00080000:3.0:1788201972.609093:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201972.609101:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201972.609176:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442825600/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201972.609208:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201972.609212:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201972.609214:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201972.609219:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1244 00000100:00080000:3.0:1788201972.609223:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201972.609227:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201972.609240:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201972.609254:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442825728/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201972.609266:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201972.609270:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201972.609272:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201972.609276:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1244 00000100:00080000:3.0:1788201972.609281:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201972.609286:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201972.609292:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201972.609297:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442825600:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788201972.609308:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442825856/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201972.609332:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201972.609337:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201972.609339:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201972.609344:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1244 00000100:00080000:3.0:1788201972.609362:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201972.609369:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201972.609375:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201972.609389:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442825984/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201972.609412:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442825728:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201972.609444:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442825856:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788201972.609461:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442825984:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201973.441069:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201974.465007:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201975.489005:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788201975.681121:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:0.0:1788201975.681175:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788201975.681193:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201975.681235:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201975.681249:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788201975.681262:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:2.0:1788201975.681290:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201972/real 1788201975] req@ffff90235268a680 x1875064442825856/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201975.681318:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442825856/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201975.681332:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201975.681338:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201975.681342:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442825856:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788201975.681351:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201972/real 1788201975] req@ffff902352689c00 x1875064442825984/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201975.681365:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442825984/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201975.681374:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201975.681378:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201975.681382:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442825984:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788201975.681391:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201972/real 1788201975] req@ffff902352689180 x1875064442825600/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201975.681404:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442825600/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201975.681420:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201975.681423:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201975.681427:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442825600:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788201975.681434:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201972/real 1788201975] req@ffff902352689500 x1875064442825728/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201975.681446:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442825728/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201975.681453:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201975.681456:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201975.681460:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442825728:192.168.201.130@tcp:38:kworker.0 00010000:00010000:2.0:1788201976.513040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201977.536990:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201977.728979:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201977.728991:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201977.728995:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201977.729007:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1249 00000100:00080000:3.0:1788201977.729019:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201977.729027:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201977.729107:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442826112/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201977.729138:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201977.729142:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201977.729144:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201977.729149:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1249 00000100:00080000:3.0:1788201977.729152:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201977.729158:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201977.729170:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201977.729183:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442826240/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201977.729193:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442826112:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201977.729196:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201977.729210:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201977.729216:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201977.729225:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1249 00000100:00080000:3.0:1788201977.729231:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201977.729237:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201977.729245:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201977.729276:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442826368/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201977.729289:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201977.729296:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201977.729305:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788201977.729306:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788201977.729327:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1249 00000100:00080000:3.0:1788201977.729336:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201977.729343:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201977.729350:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201977.729370:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201977/real 1788201977] req@ffff90235268b800 x1875064442826112/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201977.729372:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442826496/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201977.729439:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442826112/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201977.729456:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201977.729462:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201977.729468:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442826112:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788201977.729501:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442826240:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788201977.729539:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201977.729553:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201977/real 1788201977] req@ffff902352688e00 x1875064442826240/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201977.729565:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442826240/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201977.729573:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201977.729576:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201977.729580:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442826240:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201977.729593:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442826368:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201977.729615:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201977.729626:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201977/real 1788201977] req@ffff90235268a300 x1875064442826368/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201977.729639:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442826368/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201977.729648:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201977.729651:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201977.729655:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442826368:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788201977.729662:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442826496:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201977.729685:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201977.729697:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201977/real 1788201977] req@ffff902352688000 x1875064442826496/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201977.729736:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442826496/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201977.729744:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201977.729747:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201977.729751:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442826496:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201978.560995:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201979.585078:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201980.609132:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201981.633056:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201982.657115:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201982.849028:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201982.849039:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201982.849044:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201982.849054:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1254 00000100:00080000:1.0:1788201982.849066:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201982.849074:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788201982.849117:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442826624/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201982.849143:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201982.849146:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201982.849147:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201982.849157:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1254 00000100:00080000:1.0:1788201982.849160:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201982.849163:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201982.849176:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201982.849188:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442826624:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788201982.849192:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442826752/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201982.849203:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201982.849205:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201982.849207:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201982.849210:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1254 00000100:00080000:1.0:1788201982.849213:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201982.849217:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201982.849222:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201982.849236:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442826880/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201982.849247:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201982.849252:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201982.849254:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201982.849258:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1254 00000100:00080000:1.0:1788201982.849262:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201982.849268:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201982.849274:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788201982.849274:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788201982.849287:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442827008/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201982.849343:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201982/real 1788201982] req@ffff90234519ea00 x1875064442826624/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201982.849367:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442826624/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201982.849377:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201982.849380:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201982.849383:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442826624:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788201982.849407:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442826752:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788201982.849428:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201982.849437:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201982/real 1788201982] req@ffff90234519fb80 x1875064442826752/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201982.849451:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442826752/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201982.849457:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201982.849459:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201982.849462:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442826752:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201982.849465:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442826880:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201982.849478:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201982.849484:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201982/real 1788201982] req@ffff90234519c700 x1875064442826880/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201982.849491:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442826880/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201982.849497:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201982.849499:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201982.849501:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442826880:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788201982.849505:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442827008:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201982.849515:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201982.849521:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201982/real 1788201982] req@ffff90234519df80 x1875064442827008/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201982.849529:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442827008/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201982.849538:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201982.849540:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201982.849543:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442827008:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201983.681108:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201984.705097:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201985.729106:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201986.753109:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201987.776994:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201987.969014:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201987.969025:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201987.969030:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201987.969041:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1259 00000100:00080000:3.0:1788201987.969051:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201987.969059:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201987.969100:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442827136/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201987.969126:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201987.969129:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201987.969130:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201987.969133:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1259 00000100:00080000:3.0:1788201987.969136:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201987.969139:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201987.969147:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201987.969158:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442827264/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201987.969165:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201987.969168:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201987.969169:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201987.969171:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1259 00000100:00080000:3.0:1788201987.969173:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788201987.969178:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442827136:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201987.969178:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201987.969184:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201987.969193:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442827392/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201987.969214:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201987.969217:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201987.969218:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201987.969220:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1259 00000100:00080000:3.0:1788201987.969240:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201987.969242:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201987.969245:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788201987.969250:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788201987.969255:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442827520/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201987.969292:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201987/real 1788201987] req@ffff90235268b100 x1875064442827136/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201987.969312:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442827136/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201987.969323:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201987.969328:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201987.969332:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442827136:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788201987.969341:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442827264:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788201987.969359:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201987.969369:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201987/real 1788201987] req@ffff902352688000 x1875064442827264/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201987.969377:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442827264/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201987.969387:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201987.969389:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201987.969391:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442827264:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201987.969409:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442827392:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201987.969420:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201987.969427:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201987/real 1788201987] req@ffff90235268a300 x1875064442827392/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201987.969434:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442827392/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201987.969440:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201987.969441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201987.969443:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442827392:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788201987.969447:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442827520:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201987.969456:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201987.969462:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201987/real 1788201987] req@ffff902352688e00 x1875064442827520/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201987.969469:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442827520/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201987.969474:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201987.969475:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201987.969478:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442827520:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201988.801045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201989.824999:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201990.849206:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201991.873054:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201992.897019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201993.088968:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201993.088976:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201993.088979:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201993.088989:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1264 00000100:00080000:3.0:1788201993.088999:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201993.089005:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788201993.089039:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442827648/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201993.089059:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201993.089061:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201993.089063:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201993.089066:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1264 00000100:00080000:3.0:1788201993.089068:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201993.089070:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201993.089079:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201993.089090:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442827776/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201993.089103:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201993.089107:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201993.089109:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788201993.089115:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442827648:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788201993.089116:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1264 00000100:00080000:3.0:1788201993.089124:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201993.089130:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201993.089135:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201993.089148:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442827904/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201993.089157:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201993.089161:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201993.089163:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201993.089167:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1264 00000100:00080000:3.0:1788201993.089171:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788201993.089175:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788201993.089177:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788201993.089184:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201993.089196:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442828032/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201993.089213:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201993/real 1788201993] req@ffff90235268b800 x1875064442827648/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201993.089229:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442827648/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201993.089237:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201993.089241:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201993.089244:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442827648:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788201993.089260:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442827776:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788201993.089274:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201993.089283:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201993/real 1788201993] req@ffff902352689500 x1875064442827776/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201993.089290:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442827776/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201993.089296:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201993.089297:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201993.089299:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442827776:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201993.089304:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442827904:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201993.089315:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201993.089321:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201993/real 1788201993] req@ffff902352689180 x1875064442827904/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201993.089328:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442827904/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201993.089334:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201993.089336:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201993.089337:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442827904:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788201993.089340:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442828032:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201993.089351:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201993.089356:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201993/real 1788201993] req@ffff902352689c00 x1875064442828032/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201993.089363:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442828032/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201993.089368:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201993.089370:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201993.089371:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442828032:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788201993.921034:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201994.945039:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201995.969055:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201996.993033:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201998.017058:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201998.209001:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201998.209013:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201998.209017:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201998.209027:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1269 00000100:00080000:1.0:1788201998.209038:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201998.209045:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788201998.209079:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442828160/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201998.209101:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201998.209104:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201998.209105:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201998.209109:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1269 00000100:00080000:1.0:1788201998.209111:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201998.209113:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201998.209121:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201998.209129:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442828288/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201998.209137:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201998.209140:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201998.209141:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788201998.209147:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442828160:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788201998.209148:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1269 00000100:00080000:1.0:1788201998.209154:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201998.209158:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201998.209161:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201998.209168:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442828416/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201998.209180:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201998.209183:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201998.209184:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201998.209187:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1269 00000100:00080000:1.0:1788201998.209189:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201998.209191:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788201998.209194:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201998.209201:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442828544/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201998.209208:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201998.209244:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201998/real 1788201998] req@ffff90234519ed80 x1875064442828160/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201998.209258:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442828160/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201998.209274:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201998.209276:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201998.209279:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442828160:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788201998.209299:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442828288:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788201998.209314:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201998.209323:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201998/real 1788201998] req@ffff90234519d880 x1875064442828288/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201998.209331:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442828288/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201998.209336:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201998.209338:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201998.209341:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442828288:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788201998.209344:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442828416:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201998.209354:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201998.209360:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201998/real 1788201998] req@ffff90234519c000 x1875064442828416/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201998.209372:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442828416/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201998.209377:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201998.209379:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201998.209380:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442828416:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788201998.209384:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442828544:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788201998.209393:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788201998.209398:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201998/real 1788201998] req@ffff90234519e300 x1875064442828544/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201998.209405:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442828544/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201998.209410:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201998.209412:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788201998.209414:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442828544:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788201999.041093:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202000.065075:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202001.089028:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202002.113168:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202003.137082:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202003.329049:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202003.329061:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202003.329067:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202003.329079:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1274 00000100:00080000:3.0:1788202003.329093:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202003.329102:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202003.329148:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442828672/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202003.329189:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202003.329196:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202003.329199:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202003.329208:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1274 00000100:00080000:3.0:1788202003.329214:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202003.329223:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202003.329239:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202003.329265:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442828800/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202003.329279:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442828672:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202003.329288:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202003.329296:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202003.329299:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202003.329305:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1274 00000100:00080000:3.0:1788202003.329310:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202003.329315:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202003.329322:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202003.329341:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442828928/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202003.329363:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202003.329367:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202003.329369:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202003.329373:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1274 00000100:00080000:3.0:1788202003.329379:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202003.329383:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202003.329389:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788202003.329399:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202003.329410:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442829056/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202003.329472:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202003/real 1788202003] req@ffff90235268a680 x1875064442828672/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202003.329497:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442828672/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202003.329516:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202003.329523:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202003.329530:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442828672:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202003.329580:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442828800:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202003.329657:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202003.329677:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202003/real 1788202003] req@ffff902352689880 x1875064442828800/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202003.329693:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442828800/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202003.329703:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202003.329707:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202003.329711:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442828800:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202003.329719:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442828928:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202003.329739:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202003.329751:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202003/real 1788202003] req@ffff902352688380 x1875064442828928/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202003.329764:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442828928/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202003.329782:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202003.329786:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202003.329789:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442828928:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202003.329796:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442829056:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202003.329812:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202003.329822:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202003/real 1788202003] req@ffff902352688700 x1875064442829056/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202003.329850:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442829056/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202003.329859:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202003.329863:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202003.329866:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442829056:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202004.161084:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202005.185048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202006.209075:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202007.232988:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202008.257043:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202008.448992:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202008.449003:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202008.449007:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202008.449019:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1279 00000100:00080000:3.0:1788202008.449030:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202008.449037:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202008.449075:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442829184/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202008.449108:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202008.449112:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202008.449114:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202008.449118:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1279 00000100:00080000:3.0:1788202008.449121:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202008.449125:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202008.449135:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202008.449148:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442829312/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202008.449157:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442829184:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202008.449164:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202008.449169:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202008.449171:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202008.449177:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1279 00000100:00080000:3.0:1788202008.449181:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202008.449185:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202008.449191:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202008.449203:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442829440/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202008.449213:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788202008.449218:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788202008.449220:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202008.449224:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202008.449229:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1279 00000100:00080000:3.0:1788202008.449234:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202008.449238:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202008.449243:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202008.449254:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442829568/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202008.449260:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202008/real 1788202008] req@ffff90235268bb80 x1875064442829184/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202008.449283:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442829184/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202008.449293:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202008.449296:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202008.449300:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442829184:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202008.449319:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442829312:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202008.449333:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202008.449340:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202008/real 1788202008] req@ffff90235268ad80 x1875064442829312/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202008.449348:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442829312/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202008.449354:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202008.449356:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202008.449358:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442829312:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202008.449361:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442829440:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202008.449371:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202008.449377:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202008/real 1788202008] req@ffff90235268b480 x1875064442829440/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202008.449384:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442829440/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202008.449389:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202008.449391:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202008.449393:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442829440:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202008.449397:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442829568:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202008.449406:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202008.449411:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202008/real 1788202008] req@ffff902352688a80 x1875064442829568/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202008.449418:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442829568/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202008.449423:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202008.449425:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202008.449427:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442829568:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202009.281052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202010.305083:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202011.329042:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202012.353086:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202013.377105:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202013.569027:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202013.569038:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202013.569042:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202013.569053:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1285 00000100:00080000:1.0:1788202013.569066:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202013.569074:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202013.569119:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442829696/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202013.569148:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202013.569161:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202013.569163:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202013.569166:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1285 00000100:00080000:1.0:1788202013.569169:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202013.569171:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202013.569188:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202013.569201:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442829824/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202013.569211:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202013.569214:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202013.569215:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202013.569219:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1285 00000100:00080000:1.0:1788202013.569222:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202013.569227:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202013.569234:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202013.569234:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442829696:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202013.569253:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442829952/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202013.569275:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202013.569278:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202013.569279:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202013.569282:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1285 00000100:00080000:1.0:1788202013.569287:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202013.569290:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202013.569294:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202013.569307:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442830080/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202013.569358:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202013.569448:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202013/real 1788202013] req@ffff90234519f800 x1875064442829696/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202013.569475:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442829696/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202013.569487:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202013.569491:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202013.569495:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442829696:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202013.569503:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442829824:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202013.569525:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202013.569542:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202013/real 1788202013] req@ffff90234519e300 x1875064442829824/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202013.569553:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442829824/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202013.569566:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202013.569572:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202013.569580:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442829824:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202013.569625:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442829952:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202013.569650:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202013.569665:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202013/real 1788202013] req@ffff90234519c000 x1875064442829952/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202013.569680:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442829952/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202013.569693:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202013.569697:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202013.569701:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442829952:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202013.569709:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442830080:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202013.569734:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202013.569750:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202013/real 1788202013] req@ffff90234519d880 x1875064442830080/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202013.569767:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442830080/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202013.569777:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202013.569781:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202013.569785:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442830080:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202014.401078:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202015.425023:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202016.449028:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202017.472994:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202018.497036:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202018.688981:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202018.688990:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202018.688993:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202018.689002:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1290 00000100:00080000:1.0:1788202018.689011:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202018.689017:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202018.689050:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442830208/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202018.689090:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202018.689093:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202018.689095:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202018.689098:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1290 00000100:00080000:1.0:1788202018.689101:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202018.689104:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202018.689112:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202018.689122:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442830336/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202018.689130:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202018.689133:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202018.689134:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202018.689137:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1290 00000100:00080000:1.0:1788202018.689140:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202018.689143:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202018.689146:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202018.689155:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442830464/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202018.689155:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442830208:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202018.689177:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202018.689180:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202018.689181:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202018.689185:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1290 00000100:00080000:1.0:1788202018.689187:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202018.689189:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202018.689193:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202018.689202:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442830592/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202018.689230:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202018.689296:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202018/real 1788202018] req@ffff90234519ed80 x1875064442830208/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202018.689310:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442830208/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202018.689319:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202018.689321:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202018.689324:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442830208:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202018.689330:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442830336:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202018.689363:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202018.689372:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202018/real 1788202018] req@ffff90234519df80 x1875064442830336/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202018.689381:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442830336/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202018.689387:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202018.689389:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202018.689391:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442830336:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202018.689409:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442830464:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202018.689421:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202018.689428:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202018/real 1788202018] req@ffff90234519c700 x1875064442830464/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202018.689436:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442830464/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202018.689441:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202018.689443:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202018.689445:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442830464:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202018.689449:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442830592:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202018.689458:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202018.689464:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202018/real 1788202018] req@ffff90234519fb80 x1875064442830592/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202018.689471:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442830592/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202018.689480:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202018.689482:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202018.689484:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442830592:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202019.521051:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202020.545053:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202021.569047:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202022.593028:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202023.617021:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202023.809022:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202023.809032:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202023.809037:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202023.809048:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1295 00000100:00080000:0.0:1788202023.809060:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202023.809067:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202023.809111:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442830720/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202023.809142:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202023.809146:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202023.809148:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202023.809153:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1295 00000100:00080000:0.0:1788202023.809157:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202023.809162:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202023.809172:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202023.809188:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442830848/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202023.809193:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442830720:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202023.809210:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202023.809215:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202023.809217:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202023.809223:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1295 00000100:00080000:0.0:1788202023.809229:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202023.809234:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202023.809239:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788202023.809247:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202023.809271:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442830976/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202023.809294:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202023.809298:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202023.809301:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202023.809306:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1295 00000100:00080000:0.0:1788202023.809310:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202023.809315:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202023.809321:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202023.809329:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202023/real 1788202023] req@ffff9023450bb480 x1875064442830720/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202023.809337:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442831104/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202023.809371:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442830720/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202023.809381:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202023.809384:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202023.809388:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442830720:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202023.809410:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442830848:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202023.809427:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202023.809436:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202023/real 1788202023] req@ffff9023450bbb80 x1875064442830848/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202023.809444:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442830848/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202023.809451:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202023.809453:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202023.809455:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442830848:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202023.809459:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442830976:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202023.809488:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202023.809495:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202023/real 1788202023] req@ffff9023450b8e00 x1875064442830976/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202023.809503:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442830976/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202023.809508:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202023.809510:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202023.809512:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442830976:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202023.809516:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442831104:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202023.809526:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202023.809532:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202023/real 1788202023] req@ffff9023450ba680 x1875064442831104/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202023.809544:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442831104/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202023.809549:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202023.809551:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202023.809564:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442831104:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202024.641011:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202025.665048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202026.689004:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202027.713094:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202028.737018:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202028.929056:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202028.929067:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202028.929072:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202028.929083:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1300 00000100:00080000:3.0:1788202028.929097:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202028.929105:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202028.929158:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064442831232/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202028.929202:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202028.929205:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202028.929207:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202028.929211:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1300 00000100:00080000:3.0:1788202028.929213:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202028.929216:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202028.929226:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202028.929240:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442831360/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202028.929251:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202028.929255:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202028.929257:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202028.929263:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1300 00000100:00100000:2.0:1788202028.929264:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442831232:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202028.929267:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202028.929275:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202028.929281:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202028.929296:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442831488/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202028.929309:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202028.929313:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202028.929314:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202028.929317:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1300 00000100:00080000:3.0:1788202028.929328:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202028.929331:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202028.929335:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788202028.929339:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202028.929347:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442831616/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202028.929388:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202028/real 1788202028] req@ffff90235268aa00 x1875064442831232/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202028.929405:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064442831232/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202028.929413:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202028.929416:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202028.929419:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442831232:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202028.929443:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442831360:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202028.929459:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202028.929467:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202028/real 1788202028] req@ffff902352688a80 x1875064442831360/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202028.929476:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442831360/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202028.929481:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202028.929482:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202028.929485:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442831360:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202028.929490:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442831488:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202028.929501:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202028.929507:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202028/real 1788202028] req@ffff90235268b480 x1875064442831488/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202028.929514:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442831488/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202028.929520:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202028.929522:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202028.929524:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442831488:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202028.929527:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442831616:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202028.929537:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202028.929542:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202028/real 1788202028] req@ffff90235268ad80 x1875064442831616/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202028.929549:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442831616/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202028.929554:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202028.929555:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202028.929558:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442831616:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202029.761135:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202030.785061:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202031.809055:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202032.833042:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202033.857044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202034.048990:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202034.049001:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202034.049005:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202034.049025:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1305 00000100:00080000:3.0:1788202034.049036:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202034.049044:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202034.049110:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442831744/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202034.049141:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202034.049145:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202034.049147:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202034.049152:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1305 00000100:00080000:3.0:1788202034.049156:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202034.049160:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202034.049171:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202034.049185:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442831872/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202034.049193:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442831744:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202034.049203:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202034.049207:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202034.049210:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202034.049215:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1305 00000100:00080000:3.0:1788202034.049219:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202034.049223:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202034.049229:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202034.049243:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442832000/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202034.049254:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788202034.049255:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788202034.049261:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202034.049264:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202034.049269:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1305 00000100:00080000:3.0:1788202034.049277:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202034.049281:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202034.049287:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202034.049294:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202034/real 1788202034] req@ffff90235268bb80 x1875064442831744/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202034.049302:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442832128/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202034.049312:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442831744/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202034.049324:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202034.049327:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202034.049330:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442831744:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202034.049349:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442831872:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202034.049363:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202034.049371:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202034/real 1788202034] req@ffff902352688700 x1875064442831872/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202034.049379:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442831872/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202034.049385:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202034.049386:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202034.049388:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442831872:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202034.049392:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442832000:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202034.049405:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202034.049411:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202034/real 1788202034] req@ffff902352688380 x1875064442832000/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202034.049418:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442832000/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202034.049423:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202034.049425:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202034.049427:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442832000:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202034.049430:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442832128:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202034.049439:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202034.049444:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202034/real 1788202034] req@ffff902352689880 x1875064442832128/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202034.049451:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442832128/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202034.049456:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202034.049458:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202034.049459:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442832128:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202034.881076:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202035.905036:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202036.929098:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202037.953084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202038.977119:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202039.169030:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202039.169041:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202039.169047:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202039.169058:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1310 00000100:00080000:3.0:1788202039.169069:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202039.169076:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202039.169109:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442832256/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202039.169134:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202039.169137:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202039.169139:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202039.169142:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1310 00000100:00080000:3.0:1788202039.169144:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202039.169147:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202039.169155:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202039.169163:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442832384/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202039.169171:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202039.169173:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202039.169175:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202039.169177:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1310 00000100:00080000:3.0:1788202039.169180:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202039.169182:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202039.169185:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202039.169191:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442832256:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202039.169194:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442832512/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202039.169210:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202039.169213:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202039.169214:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202039.169216:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1310 00000100:00080000:3.0:1788202039.169219:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202039.169221:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202039.169224:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202039.169231:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442832640/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202039.169290:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442832384:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202039.169328:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442832512:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202039.169350:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442832640:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202040.001302:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202041.025011:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202042.049023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788202042.241479:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:0.0:1788202042.241546:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788202042.241573:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788202042.241638:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788202042.241652:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788202042.241663:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:1.0:1788202042.241873:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202039/real 1788202042] req@ffff902352689180 x1875064442832512/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202042.241955:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442832512/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202042.241971:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202042.241977:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202042.241982:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442832512:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202042.241993:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202039/real 1788202042] req@ffff902352689500 x1875064442832640/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202042.242006:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442832640/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202042.242014:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202042.242019:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202042.242023:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442832640:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202042.242033:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202039/real 1788202042] req@ffff90235268a680 x1875064442832256/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202042.242051:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442832256/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202042.242060:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202042.242065:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202042.242069:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442832256:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202042.242077:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202039/real 1788202042] req@ffff902352689c00 x1875064442832384/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202042.242104:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442832384/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202042.242113:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202042.242117:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202042.242121:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442832384:192.168.201.130@tcp:38:kworker.0 00010000:00010000:2.0:1788202043.073019:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202044.097224:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202044.288986:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202044.288995:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202044.288998:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202044.289007:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1315 00000100:00080000:3.0:1788202044.289017:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202044.289023:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202044.289055:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442832768/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202044.289078:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202044.289080:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202044.289082:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202044.289085:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1315 00000100:00080000:3.0:1788202044.289087:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202044.289090:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202044.289099:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202044.289112:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442832768:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202044.289114:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442832896/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202044.289130:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202044.289133:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202044.289135:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202044.289139:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1315 00000100:00080000:3.0:1788202044.289143:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202044.289148:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202044.289154:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202044.289166:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442833024/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202044.289173:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788202044.289184:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202044.289189:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202044.289191:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202044.289196:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1315 00000100:00080000:3.0:1788202044.289201:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202044.289206:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202044.289210:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202044/real 1788202044] req@ffff90235268b800 x1875064442832768/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788202044.289213:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202044.289232:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442833152/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202044.289239:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442832768/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202044.289256:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202044.289261:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202044.289266:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442832768:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202044.289292:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442832896:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202044.289317:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202044.289337:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202044/real 1788202044] req@ffff902352688e00 x1875064442832896/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202044.289352:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442832896/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202044.289360:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202044.289364:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202044.289367:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442832896:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202044.289374:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442833024:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202044.289398:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202044.289410:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202044/real 1788202044] req@ffff90235268a300 x1875064442833024/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202044.289430:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442833024/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202044.289442:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202044.289445:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202044.289449:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442833024:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202044.289455:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442833152:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202044.289475:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202044.289485:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202044/real 1788202044] req@ffff902352688000 x1875064442833152/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202044.289496:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442833152/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202044.289503:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202044.289506:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202044.289509:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442833152:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202045.121048:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202046.145001:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202047.169049:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202048.193068:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202049.217072:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202049.408993:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202049.409004:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202049.409010:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202049.409021:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1320 00000100:00080000:3.0:1788202049.409034:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202049.409042:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202049.409078:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442833280/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202049.409127:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202049.409131:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202049.409133:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202049.409139:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1320 00000100:00080000:3.0:1788202049.409143:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202049.409148:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202049.409157:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202049.409175:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442833408/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202049.409185:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442833280:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202049.409193:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202049.409198:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202049.409200:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202049.409206:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1320 00000100:00080000:3.0:1788202049.409210:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202049.409215:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202049.409221:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202049.409236:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442833536/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202049.409252:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202049.409292:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202049/real 1788202049] req@ffff90235268b100 x1875064442833280/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202049.409318:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442833280/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202049.409323:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202049.409331:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202049.409331:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202049.409334:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202049.409337:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202049.409342:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442833280:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202049.409361:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442833408:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202049.409379:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202049.409388:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202049/real 1788202049] req@ffff902352688000 x1875064442833408/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202049.409400:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442833408/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202049.409409:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202049.409413:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202049.409417:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442833408:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202049.409425:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442833536:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202049.409450:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788202049.409459:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1320 00000100:00080000:3.0:1788202049.409466:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788202049.409467:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202049/real 1788202049] req@ffff90235268a300 x1875064442833536/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202049.409475:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202049.409487:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202049.409494:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442833536/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202049.409508:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202049.409511:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202049.409513:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442833536:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202049.409557:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442833664/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202049.409604:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442833664:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202049.409623:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202049.409632:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202049/real 1788202049] req@ffff902352688e00 x1875064442833664/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202049.409640:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442833664/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202049.409645:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202049.409648:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202049.409650:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442833664:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202050.242490:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202051.265146:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202052.289074:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202053.313104:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202054.337074:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202054.529013:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202054.529025:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202054.529029:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202054.529041:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1326 00000100:00080000:0.0:1788202054.529052:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202054.529060:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202054.529095:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442833792/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202054.529124:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202054.529129:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202054.529130:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202054.529133:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1326 00000100:00080000:0.0:1788202054.529136:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202054.529139:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202054.529148:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202054.529160:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442833920/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202054.529173:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788202054.529175:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442833792:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202054.529178:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202054.529184:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202054.529191:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1326 00000100:00080000:0.0:1788202054.529196:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202054.529201:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202054.529207:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202054.529221:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442834048/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202054.529232:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788202054.529237:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788202054.529241:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202054.529245:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202054.529251:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1326 00000100:00080000:0.0:1788202054.529257:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202054.529262:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202054.529268:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202054.529279:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202054/real 1788202054] req@ffff9023450b9500 x1875064442833792/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202054.529283:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442834176/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202054.529315:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442833792/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202054.529329:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202054.529335:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202054.529340:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442833792:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202054.529364:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442833920:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202054.529389:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202054.529415:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202054/real 1788202054] req@ffff9023450b8a80 x1875064442833920/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202054.529425:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442833920/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202054.529431:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202054.529433:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202054.529435:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442833920:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202054.529439:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442834048:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202054.529454:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202054.529461:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202054/real 1788202054] req@ffff9023450bb800 x1875064442834048/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202054.529468:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442834048/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202054.529474:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202054.529475:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202054.529477:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442834048:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202054.529481:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442834176:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202054.529490:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202054.529496:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202054/real 1788202054] req@ffff9023450b8380 x1875064442834176/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202054.529503:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442834176/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202054.529513:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202054.529514:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202054.529516:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442834176:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202055.361071:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202056.385025:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202057.409056:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202058.433071:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202059.457043:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202059.648999:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202059.649010:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202059.649014:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202059.649025:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1331 00000100:00080000:1.0:1788202059.649036:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202059.649044:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202059.649086:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442834304/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202059.649117:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202059.649122:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202059.649124:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202059.649129:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1331 00000100:00080000:1.0:1788202059.649134:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202059.649139:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202059.649148:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442834304:192.168.201.130@tcp:250:kworker.0 00000008:00100000:1.0:1788202059.649150:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202059.649169:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442834432/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202059.649182:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202059.649186:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202059.649188:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202059.649192:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1331 00000100:00080000:1.0:1788202059.649197:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202059.649202:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:0.0:1788202059.649207:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:1.0:1788202059.649220:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202059.649236:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442834560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202059.649248:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202059.649251:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202059.649253:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202059.649258:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1331 00000100:00080000:1.0:1788202059.649263:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788202059.649269:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202059/real 1788202059] req@ffff90234519ea00 x1875064442834304/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202059.649270:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202059.649279:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202059.649295:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442834688/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202059.649305:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442834304/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202059.649318:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202059.649322:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202059.649325:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442834304:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202059.649347:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442834432:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202059.649365:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202059.649375:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202059/real 1788202059] req@ffff90234519c380 x1875064442834432/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202059.649383:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442834432/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202059.649389:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202059.649391:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202059.649393:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442834432:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202059.649397:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442834560:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202059.649408:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202059.649414:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202059/real 1788202059] req@ffff90234519f480 x1875064442834560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202059.649422:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442834560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202059.649427:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202059.649428:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202059.649430:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442834560:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202059.649433:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442834688:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202059.649442:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202059.649448:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202059/real 1788202059] req@ffff90234519dc00 x1875064442834688/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202059.649455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442834688/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202059.649460:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202059.649462:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202059.649463:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442834688:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202060.481073:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202061.505029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202062.529031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202063.553002:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202064.577021:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202064.768977:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202064.768989:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202064.768993:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202064.769001:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1336 00000100:00080000:3.0:1788202064.769010:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202064.769016:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202064.769048:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442834816/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202064.769072:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202064.769074:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202064.769076:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202064.769079:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1336 00000100:00080000:3.0:1788202064.769081:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202064.769084:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202064.769093:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202064.769101:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442834944/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202064.769110:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788202064.769113:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442834816:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202064.769113:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202064.769117:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202064.769129:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1336 00000100:00080000:3.0:1788202064.769131:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202064.769134:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202064.769138:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202064.769148:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442835072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202064.769155:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202064.769158:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202064.769159:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202064.769162:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1336 00000100:00080000:3.0:1788202064.769164:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202064.769166:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202064.769169:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788202064.769174:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202064.769178:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442835200/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202064.769210:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202064/real 1788202064] req@ffff90235268b800 x1875064442834816/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202064.769230:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442834816/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202064.769238:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202064.769240:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202064.769244:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442834816:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202064.769262:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442834944:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202064.769277:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202064.769285:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202064/real 1788202064] req@ffff902352689c00 x1875064442834944/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202064.769293:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442834944/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202064.769298:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202064.769300:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202064.769302:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442834944:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202064.769306:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442835072:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202064.769317:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202064.769323:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202064/real 1788202064] req@ffff90235268a680 x1875064442835072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202064.769330:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442835072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202064.769340:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202064.769342:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202064.769344:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442835072:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202064.769347:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442835200:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202064.769356:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202064.769362:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202064/real 1788202064] req@ffff902352689500 x1875064442835200/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202064.769369:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442835200/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202064.769374:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202064.769376:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202064.769378:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442835200:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202065.601006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202066.625072:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202067.649044:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202068.673058:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202069.697036:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202069.888977:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202069.888986:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202069.888990:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202069.889000:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1341 00000100:00080000:3.0:1788202069.889010:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202069.889016:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202069.889052:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442835328/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202069.889071:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202069.889074:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202069.889075:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202069.889078:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1341 00000100:00080000:3.0:1788202069.889080:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202069.889083:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202069.889090:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202069.889099:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442835456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202069.889107:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202069.889109:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202069.889111:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788202069.889116:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442835328:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202069.889117:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1341 00000100:00080000:3.0:1788202069.889122:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202069.889126:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202069.889130:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202069.889138:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442835584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202069.889150:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202069.889152:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202069.889154:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202069.889156:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1341 00000100:00080000:3.0:1788202069.889158:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202069.889160:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202069.889163:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202069.889169:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442835712/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202069.889198:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202069.889233:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202069/real 1788202069] req@ffff902352689180 x1875064442835328/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202069.889246:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442835328/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202069.889254:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202069.889257:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202069.889260:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442835328:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202069.889278:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442835456:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202069.889295:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202069.889303:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202069/real 1788202069] req@ffff902352689880 x1875064442835456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202069.889311:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442835456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202069.889317:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202069.889318:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202069.889320:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442835456:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202069.889325:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442835584:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202069.889335:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202069.889341:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202069/real 1788202069] req@ffff902352688380 x1875064442835584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202069.889348:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442835584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202069.889353:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202069.889355:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202069.889357:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442835584:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202069.889361:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442835712:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202069.889370:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202069.889376:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202069/real 1788202069] req@ffff902352688700 x1875064442835712/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202069.889383:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442835712/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202069.889388:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202069.889390:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202069.889392:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442835712:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202070.721046:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202071.745036:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202072.769006:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202073.793064:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202074.817061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202075.008982:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202075.008991:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202075.008994:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202075.009003:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1346 00000100:00080000:3.0:1788202075.009013:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202075.009018:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202075.009056:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442835840/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202075.009086:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202075.009090:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202075.009092:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202075.009097:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1346 00000100:00080000:3.0:1788202075.009101:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202075.009112:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202075.009126:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442835840:192.168.201.130@tcp:250:kworker.0 00000008:00100000:3.0:1788202075.009133:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202075.009151:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442835968/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202075.009160:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202075.009163:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202075.009165:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202075.009168:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1346 00000100:00080000:3.0:1788202075.009171:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202075.009174:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202075.009179:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202075.009189:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442836096/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202075.009195:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788202075.009201:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202075.009205:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202075.009206:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202075.009210:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1346 00000100:00080000:3.0:1788202075.009212:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202075.009215:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202075.009219:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202075.009227:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442836224/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202075.009243:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202075/real 1788202075] req@ffff90235268bb80 x1875064442835840/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202075.009258:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442835840/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202075.009266:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202075.009269:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202075.009272:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442835840:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202075.009290:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442835968:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202075.009307:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202075.009316:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202075/real 1788202075] req@ffff90235268ad80 x1875064442835968/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202075.009323:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442835968/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202075.009336:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202075.009349:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202075.009353:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442835968:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202075.009369:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442836096:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202075.009393:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202075.009405:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202075/real 1788202075] req@ffff90235268b480 x1875064442836096/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202075.009413:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442836096/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202075.009419:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202075.009422:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202075.009424:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442836096:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202075.009430:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442836224:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202075.009450:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202075.009463:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202075/real 1788202075] req@ffff902352688a80 x1875064442836224/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202075.009476:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442836224/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202075.009484:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202075.009487:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202075.009501:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442836224:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202075.841053:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202076.865041:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202077.889138:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202078.913106:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202079.937085:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202080.129003:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202080.129013:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202080.129018:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202080.129029:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1351 00000100:00080000:3.0:1788202080.129041:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202080.129048:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202080.129092:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064442836352/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202080.129122:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202080.129125:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202080.129126:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202080.129129:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1351 00000100:00080000:3.0:1788202080.129131:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202080.129133:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202080.129142:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202080.129152:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442836480/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202080.129159:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202080.129161:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202080.129162:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202080.129165:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1351 00000100:00080000:3.0:1788202080.129167:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202080.129170:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202080.129172:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202080.129178:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442836608/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202080.129189:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202080.129192:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202080.129195:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788202080.129197:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442836352:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202080.129198:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1351 00000100:00080000:3.0:1788202080.129201:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202080.129206:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202080.129210:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202080.129215:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442836736/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202080.129295:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202080.129346:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202080/real 1788202080] req@ffff90235268aa00 x1875064442836352/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202080.129365:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064442836352/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202080.129376:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202080.129380:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202080.129384:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442836352:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202080.129392:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442836480:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202080.129415:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202080.129428:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202080/real 1788202080] req@ffff902352688a80 x1875064442836480/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202080.129439:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442836480/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202080.129447:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202080.129451:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202080.129454:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442836480:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202080.129479:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442836608:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202080.129497:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202080.129508:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202080/real 1788202080] req@ffff90235268b480 x1875064442836608/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202080.129520:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442836608/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202080.129528:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202080.129531:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202080.129534:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442836608:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202080.129539:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442836736:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202080.129556:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202080.129566:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202080/real 1788202080] req@ffff90235268ad80 x1875064442836736/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202080.129577:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442836736/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202080.129584:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202080.129587:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202080.129600:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442836736:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202080.961026:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202081.985096:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202083.009004:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202084.033099:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202085.057027:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202085.249003:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202085.249012:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202085.249015:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202085.249024:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1356 00000100:00080000:3.0:1788202085.249033:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202085.249039:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202085.249074:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442836864/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202085.249102:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202085.249105:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202085.249107:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202085.249111:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1356 00000100:00080000:3.0:1788202085.249113:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202085.249117:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202085.249126:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202085.249136:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442836992/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202085.249147:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202085.249150:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202085.249152:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788202085.249159:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442836864:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202085.249159:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1356 00000100:00080000:3.0:1788202085.249162:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202085.249168:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202085.249172:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202085.249193:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442837120/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202085.249204:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202085.249208:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202085.249210:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202085.249214:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1356 00000100:00080000:3.0:1788202085.249217:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788202085.249222:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788202085.249222:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202085.249230:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202085.249241:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442837248/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202085.249278:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202085/real 1788202085] req@ffff90235268bb80 x1875064442836864/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202085.249299:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442836864/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202085.249309:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202085.249312:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202085.249315:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442836864:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202085.249321:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442836992:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202085.249340:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202085.249348:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202085/real 1788202085] req@ffff902352688700 x1875064442836992/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202085.249356:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442836992/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202085.249361:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202085.249363:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202085.249365:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442836992:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202085.249379:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442837120:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202085.249390:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202085.249396:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202085/real 1788202085] req@ffff902352688380 x1875064442837120/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202085.249404:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442837120/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202085.249410:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202085.249413:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202085.249415:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442837120:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202085.249419:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442837248:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202085.249429:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202085.249436:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202085/real 1788202085] req@ffff902352689880 x1875064442837248/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202085.249444:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442837248/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202085.249449:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202085.249451:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202085.249453:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442837248:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202086.081027:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202087.105041:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202088.129115:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202089.153033:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202090.177087:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202090.368988:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202090.368998:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202090.369003:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202090.369015:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1361 00000100:00080000:2.0:1788202090.369028:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202090.369036:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202090.369077:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442837376/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202090.369112:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202090.369116:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202090.369118:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202090.369122:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1361 00000100:00080000:2.0:1788202090.369126:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202090.369130:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202090.369140:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202090.369151:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442837504/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202090.369161:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788202090.369164:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442837376:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202090.369165:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202090.369168:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202090.369173:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1361 00000100:00080000:2.0:1788202090.369177:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202090.369184:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202090.369189:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202090.369199:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442837632/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202090.369207:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202090.369210:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202090.369212:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202090.369215:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1361 00000100:00080000:2.0:1788202090.369219:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202090.369223:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202090.369227:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202090.369237:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442837760/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202090.369260:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202090.369310:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202090/real 1788202090] req@ffff90234598df80 x1875064442837376/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202090.369326:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442837376/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202090.369334:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202090.369338:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202090.369342:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442837376:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202090.369369:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442837504:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202090.369391:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202090.369402:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202090/real 1788202090] req@ffff90234598f100 x1875064442837504/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202090.369414:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442837504/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202090.369421:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202090.369424:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202090.369426:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442837504:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202090.369430:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442837632:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202090.369443:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202090.369449:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202090/real 1788202090] req@ffff90234598ed80 x1875064442837632/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202090.369456:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442837632/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202090.369461:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202090.369463:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202090.369465:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442837632:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202090.369468:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442837760:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202090.369476:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202090.369481:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202090/real 1788202090] req@ffff90234598fb80 x1875064442837760/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202090.369487:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442837760/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202090.369496:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202090.369498:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202090.369500:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442837760:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202091.201058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202092.225090:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202093.249109:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202094.273112:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202095.297064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202095.489054:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202095.489063:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202095.489066:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202095.489075:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1367 00000100:00080000:1.0:1788202095.489084:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202095.489091:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202095.489124:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442837888/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202095.489153:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202095.489157:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202095.489159:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202095.489162:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1367 00000100:00080000:1.0:1788202095.489165:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202095.489169:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202095.489177:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202095.489200:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442838016/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202095.489204:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442837888:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202095.489219:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202095.489222:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202095.489224:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202095.489229:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1367 00000100:00080000:1.0:1788202095.489232:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202095.489236:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202095.489241:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202095.489250:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442838144/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202095.489265:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788202095.489271:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788202095.489271:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202095.489275:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202095.489280:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1367 00000100:00080000:1.0:1788202095.489284:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202095.489288:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202095.489292:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202095.489302:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442838272/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202095.489316:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202095/real 1788202095] req@ffff90234519f100 x1875064442837888/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202095.489336:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442837888/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202095.489346:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202095.489350:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202095.489353:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442837888:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202095.489374:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442838016:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202095.489394:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202095.489405:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202095/real 1788202095] req@ffff90234519dc00 x1875064442838016/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202095.489416:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442838016/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202095.489424:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202095.489427:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202095.489431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442838016:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202095.489436:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442838144:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202095.489452:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202095.489460:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202095/real 1788202095] req@ffff90234519f480 x1875064442838144/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202095.489471:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442838144/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202095.489479:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202095.489482:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202095.489485:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442838144:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202095.489490:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442838272:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202095.489504:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202095.489513:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202095/real 1788202095] req@ffff90234519c380 x1875064442838272/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202095.489523:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442838272/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202095.489529:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202095.489532:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202095.489535:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442838272:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202096.321061:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202097.345100:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202098.369061:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202099.393025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202100.417011:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202100.608964:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202100.608972:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202100.608975:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202100.608982:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1372 00000100:00080000:2.0:1788202100.608990:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202100.608996:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202100.609027:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442838400/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202100.609047:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202100.609049:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202100.609050:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202100.609053:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1372 00000100:00080000:2.0:1788202100.609056:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202100.609059:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202100.609066:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202100.609077:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442838400:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202100.609077:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442838528/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202100.609092:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202100.609096:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202100.609100:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202100.609104:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1372 00000100:00080000:2.0:1788202100.609107:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202100.609111:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202100.609115:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202100.609126:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442838656/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202100.609133:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788202100.609140:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202100.609144:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202100.609146:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202100.609150:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1372 00000100:00080000:2.0:1788202100.609155:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202100.609158:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202100.609163:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202100.609174:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442838784/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202100.609178:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202100/real 1788202100] req@ffff90234598fb80 x1875064442838400/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202100.609199:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442838400/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202100.609207:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202100.609210:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202100.609212:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442838400:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202100.609230:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442838528:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202100.609246:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202100.609254:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202100/real 1788202100] req@ffff90234598ed80 x1875064442838528/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202100.609262:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442838528/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202100.609266:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202100.609268:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202100.609270:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442838528:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202100.609273:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442838656:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202100.609283:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202100.609288:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202100/real 1788202100] req@ffff90234598f100 x1875064442838656/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202100.609294:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442838656/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202100.609303:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202100.609305:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202100.609307:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442838656:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202100.609310:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442838784:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202100.609318:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202100.609323:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202100/real 1788202100] req@ffff90234598df80 x1875064442838784/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202100.609329:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442838784/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202100.609333:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202100.609335:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202100.609336:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442838784:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202101.441072:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202102.464998:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202103.489045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202104.513043:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202105.537062:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202105.729006:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202105.729019:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202105.729024:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202105.729036:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1377 00000100:00080000:2.0:1788202105.729050:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202105.729058:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202105.729104:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442838912/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202105.729130:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202105.729133:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202105.729134:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202105.729137:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1377 00000100:00080000:2.0:1788202105.729140:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202105.729142:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202105.729157:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202105.729171:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d500 x1875064442839040/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202105.729183:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202105.729186:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202105.729188:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788202105.729192:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442838912:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202105.729194:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1377 00000100:00080000:2.0:1788202105.729202:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202105.729208:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202105.729214:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202105.729227:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442839168/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202105.729241:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202105.729244:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202105.729245:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202105.729248:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1377 00000100:00080000:2.0:1788202105.729251:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202105.729253:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202105.729257:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202105.729274:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442839296/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202105.729312:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442839040:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202105.729341:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442839168:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202105.729360:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442839296:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202106.561035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202107.585504:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202108.609019:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202108.801130:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:3.0:1788202108.801181:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202108.801201:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:3.0:1788202108.801243:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:3.0:1788202108.801268:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:3.0:1788202108.801278:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788202108.801293:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202105/real 1788202108] req@ffff90234598d500 x1875064442839040/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202108.801319:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d500 x1875064442839040/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202108.801331:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202108.801336:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202108.801340:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442839040:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202108.801349:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202105/real 1788202108] req@ffff90234598d180 x1875064442839168/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202108.801363:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442839168/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202108.801372:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202108.801376:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202108.801380:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442839168:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202108.801387:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202105/real 1788202108] req@ffff90234598c380 x1875064442839296/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202108.801400:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442839296/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202108.801409:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202108.801413:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202108.801417:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442839296:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202108.801424:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202105/real 1788202108] req@ffff90234598f480 x1875064442838912/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202108.801438:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442838912/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202108.801446:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202108.801450:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202108.801454:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442838912:192.168.201.130@tcp:250:kworker.0 00010000:00010000:2.0:1788202109.633003:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202110.657061:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202110.849038:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202110.849049:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202110.849053:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202110.849063:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1382 00000100:00080000:3.0:1788202110.849073:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202110.849086:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202110.849135:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442839424/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202110.849162:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202110.849166:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202110.849168:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202110.849173:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1382 00000100:00080000:3.0:1788202110.849177:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202110.849182:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202110.849191:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202110.849207:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442839552/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202110.849217:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202110.849220:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202110.849222:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202110.849226:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1382 00000100:00080000:3.0:1788202110.849230:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202110.849234:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202110.849239:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202110.849250:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442839680/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202110.849260:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202110.849264:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202110.849266:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202110.849270:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1382 00000100:00080000:3.0:1788202110.849273:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202110.849277:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202110.849282:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202110.849285:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442839424:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202110.849295:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442839808/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202110.849378:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202110.849425:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202110/real 1788202110] req@ffff902352689180 x1875064442839424/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202110.849446:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442839424/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202110.849459:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202110.849464:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202110.849468:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442839424:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202110.849476:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442839552:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202110.849502:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202110.849518:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202110/real 1788202110] req@ffff902352689500 x1875064442839552/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202110.849532:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442839552/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202110.849541:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202110.849545:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202110.849549:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442839552:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202110.849556:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442839680:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202110.849578:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202110.849591:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202110/real 1788202110] req@ffff90235268a680 x1875064442839680/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202110.849604:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442839680/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202110.849613:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202110.849617:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202110.849621:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442839680:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202110.849649:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442839808:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202110.849672:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202110.849685:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202110/real 1788202110] req@ffff902352689c00 x1875064442839808/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202110.849699:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442839808/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202110.849706:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202110.849710:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202110.849713:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442839808:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202111.681050:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202112.705087:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202113.729030:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202114.752988:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202115.777097:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202115.968997:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202115.969013:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202115.969018:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202115.969031:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1387 00000100:00080000:2.0:1788202115.969046:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202115.969054:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202115.969107:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442839936/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202115.969142:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202115.969151:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202115.969153:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202115.969158:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1387 00000100:00080000:2.0:1788202115.969162:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202115.969167:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202115.969179:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202115.969191:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442839936:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202115.969196:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442840064/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202115.969208:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202115.969211:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202115.969213:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202115.969217:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1387 00000100:00080000:2.0:1788202115.969221:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202115.969225:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202115.969239:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202115.969252:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442840192/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202115.969262:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202115.969266:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202115.969267:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202115.969271:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1387 00000100:00080000:2.0:1788202115.969275:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202115.969278:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:0.0:1788202115.969284:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:2.0:1788202115.969284:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202115.969298:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442840320/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202115.969329:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202115/real 1788202115] req@ffff90234598e300 x1875064442839936/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202115.969346:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442839936/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202115.969354:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202115.969357:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202115.969360:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442839936:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202115.969383:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442840064:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202115.969399:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202115.969407:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202115/real 1788202115] req@ffff90234598ce00 x1875064442840064/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202115.969414:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442840064/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202115.969420:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202115.969422:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202115.969424:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442840064:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202115.969427:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442840192:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202115.969436:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202115.969441:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202115/real 1788202115] req@ffff90234598c700 x1875064442840192/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202115.969448:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442840192/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202115.969457:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202115.969459:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202115.969461:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442840192:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202115.969463:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442840320:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202115.969472:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202115.969477:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202115/real 1788202115] req@ffff90234598dc00 x1875064442840320/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202115.969483:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442840320/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202115.969487:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202115.969489:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202115.969490:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442840320:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202116.801042:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202117.825065:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202118.849065:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202119.873055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202120.897129:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202121.089056:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202121.089072:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202121.089077:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202121.089103:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1392 00000100:00080000:2.0:1788202121.089117:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202121.089127:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202121.089175:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442840448/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202121.089215:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202121.089219:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202121.089222:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202121.089228:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1392 00000100:00080000:2.0:1788202121.089233:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202121.089240:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202121.089252:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202121.089257:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442840448:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202121.089272:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442840576/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202121.089286:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202121.089291:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202121.089294:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202121.089298:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1392 00000100:00080000:2.0:1788202121.089303:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202121.089309:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202121.089315:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202121.089333:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064442840704/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202121.089345:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202121.089349:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202121.089351:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202121.089357:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1392 00000800:00000100:3.0:1788202121.089363:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788202121.089366:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202121.089374:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202121.089380:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202121.089394:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064442840832/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202121.089501:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202121/real 1788202121] req@ffff90234598f800 x1875064442840448/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202121.089526:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442840448/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202121.089686:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202121.089692:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202121.089699:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442840448:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202121.089774:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442840576:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202121.089804:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202121.089828:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202121/real 1788202121] req@ffff90234598d880 x1875064442840576/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202121.089844:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442840576/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202121.089853:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202121.089857:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202121.089861:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442840576:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202121.089868:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442840704:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202121.089938:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202121.089953:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202121/real 1788202121] req@ffff90234598e680 x1875064442840704/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202121.089967:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064442840704/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202121.089986:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202121.089990:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202121.089994:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442840704:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202121.090000:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442840832:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202121.090022:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202121.090034:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202121/real 1788202121] req@ffff90234598ca80 x1875064442840832/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202121.090048:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064442840832/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202121.090057:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202121.090061:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202121.090065:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442840832:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202121.921043:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202122.945061:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202123.969030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202124.993061:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202126.017085:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202126.209000:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202126.209008:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202126.209011:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202126.209021:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1397 00000100:00080000:0.0:1788202126.209031:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202126.209037:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202126.209071:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442840960/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202126.209093:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202126.209096:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202126.209098:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202126.209101:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1397 00000100:00080000:0.0:1788202126.209103:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202126.209106:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202126.209113:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202126.209125:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442841088/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202126.209136:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202126.209140:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202126.209142:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202126.209147:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1397 00000100:00100000:3.0:1788202126.209152:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442840960:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202126.209154:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202126.209160:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202126.209181:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202126.209197:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442841216/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202126.209208:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202126.209211:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202126.209212:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202126.209215:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1397 00000100:00080000:0.0:1788202126.209219:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202126.209222:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202126.209226:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202126.209232:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202126.209238:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442841344/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202126.209287:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202126/real 1788202126] req@ffff9023450b8700 x1875064442840960/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202126.209311:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442840960/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202126.209321:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202126.209325:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202126.209329:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442840960:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202126.209336:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442841088:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202126.209360:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202126.209373:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202126/real 1788202126] req@ffff9023450b9c00 x1875064442841088/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202126.209386:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442841088/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202126.209400:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202126.209404:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202126.209407:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442841088:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202126.209430:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442841216:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202126.209449:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202126.209461:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202126/real 1788202126] req@ffff9023450b9f80 x1875064442841216/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202126.209506:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442841216/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202126.209515:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202126.209518:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202126.209520:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442841216:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202126.209525:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442841344:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202126.209541:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202126.209550:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202126/real 1788202126] req@ffff9023450bad80 x1875064442841344/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202126.209558:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442841344/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202126.209563:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202126.209565:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202126.209567:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442841344:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202127.041021:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202128.065038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202129.089083:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202130.113055:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202131.136993:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202131.328984:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202131.328992:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202131.328996:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202131.329005:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1402 00000100:00080000:2.0:1788202131.329014:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202131.329020:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202131.329055:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064442841472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202131.329079:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202131.329082:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202131.329083:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202131.329086:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1402 00000100:00080000:2.0:1788202131.329089:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202131.329091:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202131.329098:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202131.329108:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064442841600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202131.329115:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202131.329118:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202131.329119:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202131.329122:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1402 00000100:00080000:2.0:1788202131.329123:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202131.329126:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202131.329130:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202131.329140:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442841728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202131.329142:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442841472:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202131.329160:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202131.329164:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202131.329166:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202131.329171:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1402 00000100:00080000:2.0:1788202131.329185:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202131.329191:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202131.329197:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202131.329211:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202131.329214:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442841856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202131.329247:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202131/real 1788202131] req@ffff90234598ca80 x1875064442841472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202131.329262:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064442841472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202131.329278:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202131.329282:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202131.329284:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442841472:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202131.329291:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442841600:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202131.329311:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202131.329320:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202131/real 1788202131] req@ffff90234598e680 x1875064442841600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202131.329328:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064442841600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202131.329334:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202131.329336:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202131.329339:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442841600:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202131.329357:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442841728:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202131.329374:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202131.329402:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202131/real 1788202131] req@ffff90234598d880 x1875064442841728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202131.329412:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442841728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202131.329418:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202131.329420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202131.329422:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442841728:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202131.329426:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442841856:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202131.329440:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202131.329451:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202131/real 1788202131] req@ffff90234598f800 x1875064442841856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202131.329463:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442841856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202131.329471:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202131.329474:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202131.329478:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442841856:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202132.161036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202133.185074:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202134.209081:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202135.233004:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202136.257089:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202136.449051:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202136.449062:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202136.449067:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202136.449078:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1407 00000100:00080000:2.0:1788202136.449092:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202136.449099:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202136.449144:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442841984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202136.449175:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202136.449178:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202136.449179:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202136.449183:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1407 00000100:00080000:2.0:1788202136.449187:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202136.449192:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202136.449204:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202136.449219:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442842112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202136.449232:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202136.449239:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788202136.449241:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442841984:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202136.449242:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202136.449250:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1407 00000100:00080000:2.0:1788202136.449255:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202136.449259:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202136.449263:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202136.449275:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442842240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202136.449282:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202136.449285:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202136.449286:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202136.449289:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1407 00000100:00080000:2.0:1788202136.449292:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202136.449323:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202136.449328:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202136.449337:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442842368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202136.449365:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202136.449415:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202136/real 1788202136] req@ffff90234598dc00 x1875064442841984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202136.449438:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442841984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202136.449452:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202136.449457:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202136.449461:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442841984:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202136.449493:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442842112:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202136.449520:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202136.449535:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202136/real 1788202136] req@ffff90234598c700 x1875064442842112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202136.449551:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442842112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202136.449576:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202136.449580:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202136.449584:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442842112:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202136.449592:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442842240:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202136.449616:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202136.449630:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202136/real 1788202136] req@ffff90234598ce00 x1875064442842240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202136.449644:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442842240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202136.449654:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202136.449658:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202136.449661:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442842240:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202136.449669:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442842368:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202136.449690:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202136.449702:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202136/real 1788202136] req@ffff90234598e300 x1875064442842368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202136.449716:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442842368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202136.449725:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202136.449728:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202136.449732:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442842368:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202137.281021:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202138.305076:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202139.329026:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202140.353072:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202141.377014:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202141.569011:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202141.569034:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202141.569041:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202141.569054:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1413 00000100:00080000:2.0:1788202141.569067:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202141.569083:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202141.569140:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442842496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202141.569189:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202141.569192:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202141.569194:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202141.569198:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1413 00000100:00080000:2.0:1788202141.569201:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202141.569229:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202141.569250:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202141.569262:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442842624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202141.569274:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202141.569280:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788202141.569280:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442842496:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202141.569284:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202141.569293:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1413 00000100:00080000:2.0:1788202141.569298:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202141.569302:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202141.569309:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202141.569339:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442842752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202141.569351:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202141.569355:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202141.569357:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202141.569362:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1413 00000100:00080000:2.0:1788202141.569366:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788202141.569368:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788202141.569374:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202141.569381:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202141.569395:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d500 x1875064442842880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202141.569421:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202141/real 1788202141] req@ffff90234598f480 x1875064442842496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202141.569445:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442842496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202141.569458:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202141.569463:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202141.569468:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442842496:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202141.569495:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442842624:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202141.569528:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202141.569542:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202141/real 1788202141] req@ffff90234598c380 x1875064442842624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202141.569557:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442842624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202141.569567:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202141.569570:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202141.569574:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442842624:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202141.569581:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442842752:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202141.569601:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202141.569611:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202141/real 1788202141] req@ffff90234598d180 x1875064442842752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202141.569625:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442842752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202141.569635:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202141.569638:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202141.569642:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442842752:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202141.569649:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442842880:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202141.569667:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202141.569677:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202141/real 1788202141] req@ffff90234598d500 x1875064442842880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202141.569690:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d500 x1875064442842880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202141.569707:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202141.569710:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202141.569714:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442842880:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202142.401085:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202143.425014:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202144.449097:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202145.473042:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202146.497072:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202146.688977:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202146.688985:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202146.688988:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202146.688996:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1418 00000100:00080000:2.0:1788202146.689006:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202146.689012:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202146.689042:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442843008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202146.689064:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202146.689067:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202146.689068:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202146.689071:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1418 00000100:00080000:2.0:1788202146.689073:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202146.689075:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202146.689083:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202146.689092:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442843136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202146.689104:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202146.689107:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202146.689108:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202146.689111:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1418 00000100:00080000:2.0:1788202146.689113:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202146.689134:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202146.689141:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202146.689145:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442843008:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202146.689154:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442843264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202146.689164:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202146.689166:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202146.689167:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202146.689170:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1418 00000100:00080000:2.0:1788202146.689173:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202146.689177:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202146.689180:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202146.689187:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442843392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202146.689234:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202146.689280:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202146/real 1788202146] req@ffff90234598df80 x1875064442843008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202146.689303:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442843008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202146.689313:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202146.689316:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202146.689319:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442843008:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202146.689326:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442843136:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202146.689343:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202146.689358:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202146/real 1788202146] req@ffff90234598f100 x1875064442843136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202146.689366:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442843136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202146.689372:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202146.689374:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202146.689378:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442843136:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202146.689413:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442843264:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202146.689436:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202146.689446:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202146/real 1788202146] req@ffff90234598ed80 x1875064442843264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202146.689455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442843264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202146.689462:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202146.689464:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202146.689467:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442843264:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202146.689471:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442843392:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202146.689484:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202146.689490:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202146/real 1788202146] req@ffff90234598fb80 x1875064442843392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202146.689497:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442843392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202146.689502:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202146.689504:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202146.689506:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442843392:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202147.521017:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202148.545053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202149.569397:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202150.593058:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202151.617064:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202151.808976:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202151.808987:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202151.808991:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202151.808999:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1423 00000100:00080000:2.0:1788202151.809010:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202151.809016:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202151.809067:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442843520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202151.809090:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202151.809094:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202151.809096:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202151.809101:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1423 00000100:00080000:2.0:1788202151.809105:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202151.809110:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202151.809121:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202151.809132:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442843520:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202151.809140:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442843648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202151.809164:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202151.809168:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202151.809170:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202151.809175:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1423 00000100:00080000:2.0:1788202151.809180:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202151.809184:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:3.0:1788202151.809190:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:2.0:1788202151.809192:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202151.809210:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442843776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202151.809221:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202151.809226:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202151.809228:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202151.809233:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1423 00000100:00080000:2.0:1788202151.809238:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788202151.809239:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202151/real 1788202151] req@ffff90234598fb80 x1875064442843520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202151.809246:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202151.809255:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202151.809267:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442843520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202151.809277:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788202151.809280:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442843904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202151.809282:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202151.809289:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442843520:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202151.809317:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442843648:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202151.809336:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202151.809345:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202151/real 1788202151] req@ffff90234598ed80 x1875064442843648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202151.809354:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442843648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202151.809360:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202151.809362:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202151.809364:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442843648:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202151.809368:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442843776:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202151.809378:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202151.809384:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202151/real 1788202151] req@ffff90234598f100 x1875064442843776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202151.809392:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442843776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202151.809401:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202151.809403:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202151.809405:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442843776:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202151.809408:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442843904:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202151.809417:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202151.809423:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202151/real 1788202151] req@ffff90234598df80 x1875064442843904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202151.809430:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442843904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202151.809434:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202151.809436:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202151.809438:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442843904:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202152.641118:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202153.665067:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202154.689027:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202155.713051:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202156.737029:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202156.928969:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202156.928978:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202156.928981:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202156.928990:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1428 00000100:00080000:2.0:1788202156.929000:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202156.929006:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202156.929039:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d500 x1875064442844032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202156.929068:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202156.929071:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202156.929073:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202156.929076:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1428 00000100:00080000:2.0:1788202156.929078:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202156.929082:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202156.929091:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202156.929104:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442844032:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202156.929108:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442844160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202156.929132:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202156.929136:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202156.929138:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202156.929143:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1428 00000100:00080000:2.0:1788202156.929154:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202156.929158:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:3.0:1788202156.929162:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:2.0:1788202156.929164:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202156.929178:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442844288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202156.929185:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202156.929188:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202156.929190:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202156.929193:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1428 00000100:00080000:2.0:1788202156.929197:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788202156.929198:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202156/real 1788202156] req@ffff90234598d500 x1875064442844032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202156.929203:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202156.929211:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202156.929222:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d500 x1875064442844032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202156.929223:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442844416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202156.929249:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202156.929254:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202156.929259:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442844032:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202156.929285:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442844160:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202156.929312:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202156.929330:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202156/real 1788202156] req@ffff90234598d180 x1875064442844160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202156.929346:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442844160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202156.929355:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202156.929360:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202156.929364:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442844160:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202156.929371:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442844288:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202156.929391:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202156.929400:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202156/real 1788202156] req@ffff90234598c380 x1875064442844288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202156.929411:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442844288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202156.929418:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202156.929421:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202156.929424:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442844288:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202156.929429:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442844416:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202156.929444:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202156.929453:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202156/real 1788202156] req@ffff90234598f480 x1875064442844416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202156.929462:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442844416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202156.929471:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202156.929475:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202156.929478:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442844416:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202157.761121:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202158.785028:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202159.809110:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202160.833056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202161.857048:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202162.049005:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202162.049015:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202162.049019:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202162.049027:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1433 00000100:00080000:0.0:1788202162.049037:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202162.049043:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202162.049080:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442844544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202162.049103:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202162.049106:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202162.049108:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202162.049111:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1433 00000100:00080000:0.0:1788202162.049113:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202162.049115:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202162.049123:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202162.049131:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442844672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202162.049145:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202162.049152:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202162.049154:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202162.049157:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1433 00000100:00080000:0.0:1788202162.049159:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202162.049162:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202162.049168:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202162.049168:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442844544:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202162.049184:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442844800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202162.049196:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202162.049200:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202162.049202:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202162.049206:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1433 00000100:00080000:0.0:1788202162.049210:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202162.049215:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202162.049220:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202162.049234:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442844928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202162.049241:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202162.049300:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202162/real 1788202162] req@ffff9023450b8000 x1875064442844544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202162.049321:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442844544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202162.049333:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202162.049338:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202162.049357:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442844544:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202162.049366:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442844672:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202162.049393:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202162.049407:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202162/real 1788202162] req@ffff9023450ba300 x1875064442844672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202162.049421:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442844672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202162.049437:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202162.049441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202162.049445:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442844672:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202162.049470:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442844800:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202162.049494:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202162.049505:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202162/real 1788202162] req@ffff9023450baa00 x1875064442844800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202162.049518:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442844800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202162.049527:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202162.049531:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202162.049545:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442844800:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202162.049553:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442844928:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202162.049573:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202162.049586:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202162/real 1788202162] req@ffff9023450b9180 x1875064442844928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202162.049598:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442844928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202162.049606:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202162.049610:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202162.049614:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442844928:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202162.881009:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202163.905035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202164.929052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202165.953770:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202166.977086:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202167.168981:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202167.168989:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202167.168992:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202167.169001:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1438 00000100:00080000:0.0:1788202167.169010:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202167.169016:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202167.169051:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442845056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202167.169076:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202167.169079:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202167.169080:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202167.169083:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1438 00000100:00080000:0.0:1788202167.169085:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202167.169088:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202167.169095:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202167.169102:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442845184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202167.169110:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202167.169112:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202167.169114:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202167.169117:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1438 00000100:00080000:0.0:1788202167.169120:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202167.169122:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202167.169128:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202167.169129:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442845056:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202167.169143:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442845312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202167.169154:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202167.169159:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202167.169161:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202167.169165:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1438 00000100:00080000:0.0:1788202167.169169:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202167.169174:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202167.169178:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202167.169190:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442845440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202167.169200:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202167.169250:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202167/real 1788202167] req@ffff9023450b9180 x1875064442845056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202167.169264:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442845056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202167.169272:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202167.169276:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202167.169280:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442845056:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202167.169286:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442845184:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202167.169303:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202167.169311:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202167/real 1788202167] req@ffff9023450baa00 x1875064442845184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202167.169319:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442845184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202167.169325:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202167.169327:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202167.169330:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442845184:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202167.169347:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442845312:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202167.169358:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202167.169372:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202167/real 1788202167] req@ffff9023450ba300 x1875064442845312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202167.169380:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442845312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202167.169386:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202167.169388:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202167.169390:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442845312:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202167.169393:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442845440:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202167.169404:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202167.169410:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202167/real 1788202167] req@ffff9023450b8000 x1875064442845440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202167.169417:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442845440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202167.169422:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202167.169424:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202167.169425:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442845440:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202168.001027:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202169.025099:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202170.049037:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202171.073098:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202172.097103:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202172.288997:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202172.289022:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202172.289028:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202172.289042:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1443 00000100:00080000:3.0:1788202172.289056:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202172.289065:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202172.289121:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442845568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202172.289156:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202172.289164:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202172.289166:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202172.289171:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1443 00000100:00080000:3.0:1788202172.289175:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202172.289179:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202172.289193:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442845568:192.168.201.130@tcp:250:kworker.0 00000008:00100000:3.0:1788202172.289194:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202172.289211:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442845696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202172.289220:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202172.289223:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202172.289224:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202172.289227:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1443 00000100:00080000:3.0:1788202172.289230:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202172.289233:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202172.289242:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202172.289251:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442845824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202172.289259:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202172.289262:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202172.289263:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202172.289268:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1443 00000100:00080000:3.0:1788202172.289271:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202172.289274:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202172.289277:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202172.289285:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442845952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202172.289331:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442845696:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202172.289372:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442845824:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202172.289398:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442845952:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202173.121461:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202174.145098:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202175.170021:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788202175.361298:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788202175.361354:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788202175.361377:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202175.361434:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202175.361450:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202175.361467:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788202175.361726:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202172/real 1788202175] req@ffff902352688e00 x1875064442845696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202175.361773:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442845696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202175.361790:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202175.361797:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202175.361802:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442845696:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202175.361813:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202172/real 1788202175] req@ffff90235268a300 x1875064442845824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202175.361827:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442845824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202175.361837:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202175.361841:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202175.361845:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442845824:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202175.361853:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202172/real 1788202175] req@ffff902352688000 x1875064442845952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202175.361865:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442845952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202175.361874:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202175.361919:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202175.361925:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442845952:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202175.361934:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202172/real 1788202175] req@ffff90235268b800 x1875064442845568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202175.361950:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442845568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202175.361959:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202175.361964:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202175.361968:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442845568:192.168.201.130@tcp:250:kworker.0 00010000:00010000:0.0:1788202176.193002:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202177.217040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202177.408986:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202177.409001:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202177.409015:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202177.409034:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1448 00000100:00080000:3.0:1788202177.409050:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202177.409060:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202177.409112:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442846080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202177.409147:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202177.409151:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202177.409154:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202177.409159:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1448 00000100:00080000:3.0:1788202177.409163:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202177.409169:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202177.409183:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202177.409198:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442846080:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202177.409201:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442846208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202177.409216:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202177.409222:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202177.409224:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202177.409229:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1448 00000100:00080000:3.0:1788202177.409233:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202177.409238:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202177.409243:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202177.409256:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442846336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202177.409268:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202177.409272:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202177.409274:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788202177.409281:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788202177.409282:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1448 00000100:00080000:3.0:1788202177.409290:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202177.409295:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202177.409300:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202177.409312:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442846464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202177.409340:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202177/real 1788202177] req@ffff90235268b100 x1875064442846080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202177.409366:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442846080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202177.409380:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202177.409385:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202177.409389:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442846080:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202177.409420:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442846208:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202177.409449:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202177.409464:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202177/real 1788202177] req@ffff90235268b800 x1875064442846208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202177.409487:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442846208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202177.409494:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202177.409497:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202177.409501:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442846208:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202177.409508:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442846336:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202177.409533:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202177.409546:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202177/real 1788202177] req@ffff902352688000 x1875064442846336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202177.409559:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442846336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202177.409568:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202177.409572:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202177.409576:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442846336:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202177.409595:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442846464:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202177.409620:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202177.409633:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202177/real 1788202177] req@ffff90235268a300 x1875064442846464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202177.409647:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442846464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202177.409662:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202177.409666:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202177.409670:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442846464:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202178.241290:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202179.265018:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202180.289017:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202181.313056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202182.337131:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202182.529045:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202182.529060:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202182.529065:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202182.529078:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1454 00000100:00080000:2.0:1788202182.529091:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202182.529101:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202182.529147:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442846592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202182.529178:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202182.529183:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202182.529185:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202182.529189:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1454 00000100:00080000:2.0:1788202182.529193:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202182.529198:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202182.529208:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202182.529220:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442846720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202182.529229:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202182.529233:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202182.529234:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202182.529238:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1454 00000100:00080000:2.0:1788202182.529242:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202182.529246:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202182.529251:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202182.529261:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442846848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202182.529270:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202182.529274:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202182.529276:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202182.529279:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1454 00000100:00080000:2.0:1788202182.529283:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202182.529286:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202182.529291:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202182.529300:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442846976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202182.529346:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442846592:192.168.201.130@tcp:250:kworker.0 00000800:00000100:0.0:1788202182.529412:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202182.529445:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202182/real 1788202182] req@ffff90234598e300 x1875064442846592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202182.529459:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442846592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202182.529467:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202182.529470:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202182.529473:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442846592:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202182.529478:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442846720:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202182.529491:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202182.529513:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202182/real 1788202182] req@ffff90234598ce00 x1875064442846720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202182.529522:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442846720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202182.529530:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202182.529534:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202182.529536:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442846720:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202182.529542:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442846848:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202182.529561:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202182.529572:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202182/real 1788202182] req@ffff90234598c700 x1875064442846848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202182.529583:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442846848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202182.529591:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202182.529594:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202182.529598:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442846848:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202182.529603:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442846976:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202182.529623:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202182.529633:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202182/real 1788202182] req@ffff90234598dc00 x1875064442846976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202182.529643:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442846976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202182.529651:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202182.529654:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202182.529657:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442846976:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202183.361060:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202184.384986:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202185.409076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202186.433046:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202187.457050:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202187.648988:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202187.648998:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202187.649002:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202187.649012:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1459 00000100:00080000:3.0:1788202187.649024:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202187.649031:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202187.649070:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442847104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202187.649096:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202187.649099:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202187.649100:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202187.649103:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1459 00000100:00080000:3.0:1788202187.649105:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202187.649108:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202187.649117:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202187.649125:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442847232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202187.649132:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202187.649135:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202187.649137:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202187.649139:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1459 00000100:00080000:3.0:1788202187.649141:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202187.649144:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202187.649148:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202187.649160:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442847360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202187.649171:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202187.649179:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788202187.649180:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442847104:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202187.649183:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202187.649192:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1459 00000100:00080000:3.0:1788202187.649196:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202187.649200:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202187.649205:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202187.649216:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442847488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202187.649267:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202187.649323:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202187/real 1788202187] req@ffff902352688e00 x1875064442847104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202187.649346:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442847104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202187.649357:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202187.649361:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202187.649365:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442847104:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202187.649373:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442847232:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202187.649400:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202187.649437:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202187/real 1788202187] req@ffff902352689c00 x1875064442847232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202187.649449:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442847232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202187.649454:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202187.649456:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202187.649458:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442847232:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202187.649476:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442847360:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202187.649490:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202187.649497:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202187/real 1788202187] req@ffff90235268a680 x1875064442847360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202187.649504:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442847360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202187.649509:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202187.649511:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202187.649513:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442847360:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202187.649516:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442847488:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202187.649525:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202187.649533:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202187/real 1788202187] req@ffff902352689500 x1875064442847488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202187.649544:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442847488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202187.649551:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202187.649554:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202187.649557:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442847488:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202188.481028:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202189.505072:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202190.529001:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202191.553034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202192.577016:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202192.768972:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202192.768979:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202192.768983:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202192.768991:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1464 00000100:00080000:3.0:1788202192.769000:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202192.769005:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202192.769032:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442847616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202192.769055:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202192.769057:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202192.769059:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202192.769061:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1464 00000100:00080000:3.0:1788202192.769064:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202192.769065:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202192.769073:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202192.769086:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442847744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202192.769096:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788202192.769101:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442847616:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202192.769103:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202192.769107:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202192.769113:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1464 00000100:00080000:3.0:1788202192.769117:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202192.769122:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202192.769133:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202192.769145:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442847872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202192.769152:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202192.769155:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202192.769157:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202192.769159:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1464 00000100:00080000:3.0:1788202192.769162:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788202192.769166:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788202192.769168:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202192.769172:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202192.769221:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202192/real 1788202192] req@ffff902352689180 x1875064442847616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202192.769230:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442848000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202192.769259:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442847616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202192.769274:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202192.769278:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202192.769281:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442847616:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202192.769304:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442847744:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202192.769354:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202192.769366:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202192/real 1788202192] req@ffff902352689880 x1875064442847744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202192.769375:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442847744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202192.769382:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202192.769385:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202192.769387:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442847744:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202192.769393:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442847872:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202192.769409:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202192.769419:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202192/real 1788202192] req@ffff902352688380 x1875064442847872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202192.769429:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442847872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202192.769436:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202192.769439:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202192.769441:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442847872:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202192.769447:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442848000:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202192.769463:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202192.769473:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202192/real 1788202192] req@ffff902352688700 x1875064442848000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202192.769483:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442848000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202192.769490:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202192.769493:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202192.769496:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442848000:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202193.601017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202194.625037:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202195.649035:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202196.673032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202197.697017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202197.888984:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202197.888994:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202197.888997:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202197.889007:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1469 00000100:00080000:1.0:1788202197.889016:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202197.889023:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202197.889058:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442848128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202197.889083:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202197.889086:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202197.889089:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202197.889093:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1469 00000100:00080000:1.0:1788202197.889097:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202197.889101:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202197.889108:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442848128:192.168.201.130@tcp:250:kworker.0 00000008:00100000:1.0:1788202197.889109:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202197.889136:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442848256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202197.889146:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202197.889150:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202197.889152:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202197.889155:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1469 00000100:00080000:1.0:1788202197.889160:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788202197.889160:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788202197.889167:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202197.889174:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202197.889197:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442848384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202197.889201:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202197/real 1788202197] req@ffff90234519ea00 x1875064442848128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202197.889221:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202197.889236:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788202197.889237:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442848128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202197.889241:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202197.889250:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202197.889251:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1469 00000100:00080000:2.0:1788202197.889257:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00080000:1.0:1788202197.889259:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788202197.889261:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442848128:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202197.889267:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202197.889272:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202197.889286:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442848256:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202197.889287:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442848512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202197.889323:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202197.889336:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202197/real 1788202197] req@ffff90234519fb80 x1875064442848256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202197.889347:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442848256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202197.889354:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202197.889357:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202197.889360:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442848256:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202197.889365:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442848384:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202197.889391:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202197.889400:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202197/real 1788202197] req@ffff90234519c700 x1875064442848384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202197.889410:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442848384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202197.889417:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202197.889420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202197.889422:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442848384:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202197.889434:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442848512:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202197.889448:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202197.889457:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202197/real 1788202197] req@ffff90234519df80 x1875064442848512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202197.889472:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442848512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202197.889479:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202197.889481:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202197.889485:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442848512:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202198.721032:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202199.745005:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202200.769090:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202201.793000:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202202.817014:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202203.009003:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202203.009017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202203.009023:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202203.009035:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1474 00000100:00080000:1.0:1788202203.009050:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202203.009058:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202203.009109:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442848640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202203.009135:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202203.009148:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202203.009150:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202203.009155:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1474 00000100:00080000:1.0:1788202203.009159:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202203.009163:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202203.009175:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202203.009189:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442848768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202203.009201:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442848640:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202203.009202:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202203.009210:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202203.009213:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202203.009218:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1474 00000100:00080000:1.0:1788202203.009221:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202203.009225:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202203.009231:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202203.009244:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442848896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202203.009254:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202203.009257:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202203.009259:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202203.009263:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1474 00000100:00080000:1.0:1788202203.009268:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788202203.009273:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788202203.009274:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202203.009282:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202203.009295:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442849024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202203.009318:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202203/real 1788202203] req@ffff90234519ed80 x1875064442848640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202203.009342:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442848640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202203.009352:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202203.009356:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202203.009360:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442848640:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202203.009389:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442848768:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202203.009413:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202203.009426:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202203/real 1788202203] req@ffff90234519d880 x1875064442848768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202203.009438:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442848768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202203.009447:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202203.009451:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202203.009454:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442848768:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202203.009460:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442848896:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202203.009478:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202203.009490:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202203/real 1788202203] req@ffff90234519c000 x1875064442848896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202203.009502:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442848896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202203.009510:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202203.009514:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202203.009517:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442848896:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202203.009523:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442849024:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202203.009541:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202203.009551:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202203/real 1788202203] req@ffff90234519e300 x1875064442849024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202203.009563:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442849024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202203.009570:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202203.009573:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202203.009577:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442849024:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202203.841098:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202204.865093:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202205.889025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202206.913049:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202207.937031:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202208.129000:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202208.129012:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202208.129017:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202208.129027:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1479 00000100:00080000:1.0:1788202208.129037:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202208.129044:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202208.129109:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442849152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202208.129139:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202208.129143:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202208.129146:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202208.129150:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1479 00000100:00080000:1.0:1788202208.129154:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202208.129158:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202208.129168:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202208.129183:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442849280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202208.129194:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788202208.129197:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442849152:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202208.129199:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202208.129203:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202208.129210:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1479 00000100:00080000:1.0:1788202208.129214:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202208.129218:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202208.129225:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202208.129237:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442849408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202208.129246:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202208.129250:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202208.129252:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202208.129256:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1479 00000100:00080000:1.0:1788202208.129261:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202208.129265:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:2.0:1788202208.129270:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:1.0:1788202208.129281:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202208.129296:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442849536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202208.129331:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202208/real 1788202208] req@ffff90234519f800 x1875064442849152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202208.129351:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442849152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202208.129364:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202208.129369:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202208.129374:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442849152:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202208.129398:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442849280:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202208.129422:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202208.129436:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202208/real 1788202208] req@ffff90234519e300 x1875064442849280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202208.129450:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442849280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202208.129459:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202208.129463:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202208.129467:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442849280:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202208.129476:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442849408:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202208.129497:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202208.129508:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202208/real 1788202208] req@ffff90234519c000 x1875064442849408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202208.129521:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442849408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202208.129529:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202208.129533:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202208.129537:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442849408:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202208.129542:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442849536:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202208.129559:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202208.129569:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202208/real 1788202208] req@ffff90234519d880 x1875064442849536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202208.129583:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442849536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202208.129591:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202208.129594:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202208.129598:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442849536:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202208.961066:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202209.985087:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202211.009045:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202212.033086:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202213.057033:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202213.248945:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202213.248953:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202213.248956:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202213.248977:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1484 00000100:00080000:1.0:1788202213.248986:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202213.248992:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202213.249025:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442849664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202213.249056:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202213.249060:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202213.249062:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202213.249067:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1484 00000100:00080000:1.0:1788202213.249070:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202213.249075:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202213.249084:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202213.249101:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442849792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202213.249114:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442849664:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202213.249115:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202213.249125:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202213.249128:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202213.249133:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1484 00000100:00080000:1.0:1788202213.249138:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202213.249143:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202213.249149:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202213.249163:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442849920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202213.249174:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202213.249178:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202213.249180:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202213.249188:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1484 00000800:00000100:2.0:1788202213.249189:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788202213.249198:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202213.249204:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202213.249210:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202213.249226:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442850048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202213.249235:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202213/real 1788202213] req@ffff90234519ed80 x1875064442849664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202213.249259:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442849664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202213.249281:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202213.249285:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202213.249288:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442849664:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202213.249310:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442849792:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202213.249328:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202213.249338:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202213/real 1788202213] req@ffff90234519df80 x1875064442849792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202213.249347:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442849792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202213.249352:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202213.249354:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202213.249356:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442849792:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202213.249360:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442849920:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202213.249371:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202213.249378:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202213/real 1788202213] req@ffff90234519c700 x1875064442849920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202213.249389:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442849920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202213.249395:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202213.249396:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202213.249398:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442849920:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202213.249401:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442850048:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202213.249410:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202213.249416:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202213/real 1788202213] req@ffff90234519fb80 x1875064442850048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202213.249423:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442850048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202213.249428:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202213.249429:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202213.249431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442850048:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202214.081046:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202215.105048:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202216.129161:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202217.153059:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202218.177076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202218.368982:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202218.368992:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202218.368995:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202218.369012:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1489 00000100:00080000:3.0:1788202218.369025:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202218.369033:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202218.369073:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442850176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202218.369104:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202218.369109:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202218.369111:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202218.369115:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1489 00000100:00080000:3.0:1788202218.369119:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202218.369122:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202218.369130:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202218.369142:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442850304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202218.369152:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442850176:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202218.369154:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202218.369160:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202218.369162:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202218.369165:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1489 00000100:00080000:3.0:1788202218.369168:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202218.369170:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202218.369173:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202218.369181:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442850432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202218.369188:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202218.369191:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202218.369192:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202218.369195:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1489 00000100:00080000:3.0:1788202218.369198:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202218.369201:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202218.369204:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202218.369210:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442850560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202218.369235:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202218.369270:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202218/real 1788202218] req@ffff90235268bb80 x1875064442850176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202218.369285:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442850176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202218.369293:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202218.369296:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202218.369299:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442850176:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202218.369320:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442850304:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202218.369344:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202218.369358:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202218/real 1788202218] req@ffff90235268ad80 x1875064442850304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202218.369372:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442850304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202218.369378:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202218.369380:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202218.369383:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442850304:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202218.369387:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442850432:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202218.369404:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202218.369412:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202218/real 1788202218] req@ffff90235268b480 x1875064442850432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202218.369420:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442850432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202218.369425:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202218.369428:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202218.369430:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442850432:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202218.369434:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442850560:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202218.369448:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202218.369472:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202218/real 1788202218] req@ffff902352688a80 x1875064442850560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202218.369486:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442850560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202218.369495:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202218.369498:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202218.369501:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442850560:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202219.201018:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202220.225060:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202221.249029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202222.273047:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202223.297049:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202223.489014:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202223.489022:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202223.489025:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202223.489034:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1495 00000100:00080000:1.0:1788202223.489044:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202223.489049:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202223.489092:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442850688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202223.489129:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202223.489132:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202223.489136:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202223.489140:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1495 00000100:00080000:1.0:1788202223.489144:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202223.489147:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202223.489156:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202223.489169:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442850816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202223.489169:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442850688:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202223.489184:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202223.489186:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202223.489188:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202223.489191:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1495 00000100:00080000:1.0:1788202223.489193:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202223.489195:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202223.489198:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202223.489208:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442850944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202223.489215:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202223.489217:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202223.489219:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202223.489221:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1495 00000100:00080000:1.0:1788202223.489223:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202223.489225:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202223.489229:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788202223.489233:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202223.489239:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442851072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202223.489272:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202223/real 1788202223] req@ffff90234519ea00 x1875064442850688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202223.489286:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442850688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202223.489298:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202223.489302:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202223.489306:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442850688:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202223.489329:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442850816:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202223.489354:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202223.489368:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202223/real 1788202223] req@ffff90234519c380 x1875064442850816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202223.489380:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442850816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202223.489388:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202223.489391:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202223.489395:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442850816:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202223.489401:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442850944:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202223.489419:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202223.489431:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202223/real 1788202223] req@ffff90234519f480 x1875064442850944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202223.489444:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442850944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202223.489453:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202223.489457:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202223.489461:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442850944:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202223.489467:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442851072:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202223.489486:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202223.489498:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202223/real 1788202223] req@ffff90234519dc00 x1875064442851072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202223.489519:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442851072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202223.489528:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202223.489531:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202223.489535:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442851072:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202224.321054:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202225.345064:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202226.369081:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202227.393036:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202228.417034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202228.608986:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202228.608998:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202228.609002:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202228.609013:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1500 00000100:00080000:0.0:1788202228.609024:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202228.609043:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202228.609083:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442851200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202228.609119:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202228.609132:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202228.609135:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202228.609145:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1500 00000100:00080000:0.0:1788202228.609151:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202228.609156:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442851200:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202228.609160:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202228.609174:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202228.609192:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442851328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202228.609204:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202228.609208:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202228.609211:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202228.609215:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1500 00000100:00080000:0.0:1788202228.609219:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202228.609223:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202228.609230:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202228.609232:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202228.609249:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442851456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202228.609258:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202228.609261:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202228.609262:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202228.609267:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1500 00000100:00080000:0.0:1788202228.609272:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202228.609272:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202228/real 1788202228] req@ffff9023450bad80 x1875064442851200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202228.609287:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202228.609293:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202228.609298:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442851200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202228.609303:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442851584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202228.609330:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202228.609337:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202228.609341:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442851200:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202228.609363:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442851328:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202228.609383:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202228.609394:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202228/real 1788202228] req@ffff9023450b9f80 x1875064442851328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202228.609408:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442851328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202228.609416:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202228.609420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202228.609424:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442851328:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202228.609430:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442851456:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202228.609452:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202228.609462:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202228/real 1788202228] req@ffff9023450b9c00 x1875064442851456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202228.609483:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442851456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202228.609493:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202228.609496:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202228.609500:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442851456:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202228.609506:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442851584:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202228.609522:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202228.609531:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202228/real 1788202228] req@ffff9023450b8700 x1875064442851584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202228.609538:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442851584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202228.609544:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202228.609546:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202228.609549:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442851584:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202229.441055:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202230.465025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202231.489065:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202232.513031:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202233.537064:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202233.729015:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202233.729025:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202233.729029:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202233.729041:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1505 00000100:00080000:1.0:1788202233.729055:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202233.729063:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202233.729114:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442851712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202233.729142:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202233.729144:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202233.729146:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202233.729149:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1505 00000100:00080000:1.0:1788202233.729152:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202233.729156:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202233.729166:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202233.729177:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442851840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202233.729183:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202233.729186:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202233.729187:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202233.729191:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1505 00000100:00080000:1.0:1788202233.729193:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202233.729196:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202233.729201:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202233.729214:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442851712:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202233.729215:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442851968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202233.729242:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202233.729246:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202233.729249:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202233.729255:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1505 00000100:00080000:1.0:1788202233.729258:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202233.729261:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202233.729266:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202233.729276:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442852096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202233.729311:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202233.729362:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202233/real 1788202233] req@ffff90234519f100 x1875064442851712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202233.729382:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442851712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202233.729391:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202233.729395:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202233.729398:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442851712:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202233.729404:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442851840:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202233.729424:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202233.729433:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202233/real 1788202233] req@ffff90234519dc00 x1875064442851840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202233.729441:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442851840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202233.729446:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202233.729449:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202233.729451:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442851840:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202233.729470:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442851968:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202233.729481:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202233.729487:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202233/real 1788202233] req@ffff90234519f480 x1875064442851968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202233.729495:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442851968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202233.729500:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202233.729502:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202233.729504:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442851968:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202233.729508:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442852096:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202233.729517:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202233.729523:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202233/real 1788202233] req@ffff90234519c380 x1875064442852096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202233.729535:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442852096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202233.729540:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202233.729541:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202233.729543:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442852096:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202234.561112:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202235.585078:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202236.609020:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202237.633111:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202238.657045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202238.849035:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202238.849050:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202238.849055:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202238.849067:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1510 00000100:00080000:1.0:1788202238.849081:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202238.849095:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202238.849144:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442852224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202238.849181:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202238.849185:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202238.849187:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202238.849192:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1510 00000100:00080000:1.0:1788202238.849196:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202238.849200:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202238.849212:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202238.849225:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442852352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202238.849234:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442852224:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202238.849250:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202238.849254:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202238.849256:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202238.849262:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1510 00000100:00080000:1.0:1788202238.849265:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202238.849270:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202238.849281:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202238.849293:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442852480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202238.849302:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202238.849306:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202238.849308:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202238.849312:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1510 00000100:00080000:1.0:1788202238.849316:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202238.849320:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202238.849325:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202238.849337:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442852608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202238.849354:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442852352:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202238.849391:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442852480:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202238.849419:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442852608:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202239.681439:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202240.705040:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202241.729026:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788202241.921208:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788202241.921284:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788202241.921310:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202241.921367:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202241.921379:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202241.921389:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:1.0:1788202241.921514:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202238/real 1788202241] req@ffff90234519df80 x1875064442852608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202241.921553:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442852608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202241.921564:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202241.921568:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202241.921573:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442852608:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202241.921581:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202238/real 1788202241] req@ffff90234519fb80 x1875064442852352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202241.921597:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442852352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202241.921605:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202241.921608:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202241.921611:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442852352:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202241.921619:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202238/real 1788202241] req@ffff90234519c700 x1875064442852480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202241.921629:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442852480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202241.921636:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202241.921639:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202241.921653:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442852480:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202241.921661:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202238/real 1788202241] req@ffff90234519ea00 x1875064442852224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202241.921673:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442852224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202241.921680:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202241.921683:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202241.921687:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442852224:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788202242.753160:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202243.776998:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202243.968980:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202243.968987:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202243.968990:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202243.968999:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1515 00000100:00080000:2.0:1788202243.969008:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202243.969014:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202243.969041:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442852736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202243.969058:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202243.969061:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202243.969062:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202243.969065:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1515 00000100:00080000:2.0:1788202243.969066:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202243.969069:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202243.969076:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202243.969083:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442852864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202243.969090:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202243.969093:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788202243.969094:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442852736:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202243.969095:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202243.969101:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1515 00000100:00080000:2.0:1788202243.969105:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202243.969107:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202243.969110:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202243.969116:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064442852992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202243.969122:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202243.969124:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202243.969126:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202243.969128:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1515 00000100:00080000:2.0:1788202243.969130:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202243.969132:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202243.969134:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202243.969140:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064442853120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202243.969161:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202243.969214:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202243/real 1788202243] req@ffff90234598f800 x1875064442852736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202243.969232:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442852736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202243.969241:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202243.969246:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202243.969250:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442852736:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202243.969280:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442852864:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202243.969301:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202243.969312:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202243/real 1788202243] req@ffff90234598d880 x1875064442852864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202243.969323:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442852864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202243.969331:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202243.969335:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202243.969338:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442852864:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202243.969343:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442852992:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202243.969374:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202243.969385:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202243/real 1788202243] req@ffff90234598e680 x1875064442852992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202243.969394:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064442852992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202243.969402:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202243.969405:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202243.969407:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442852992:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202243.969412:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442853120:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202243.969444:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202243.969454:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202243/real 1788202243] req@ffff90234598ca80 x1875064442853120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202243.969464:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064442853120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202243.969471:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202243.969474:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202243.969477:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442853120:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202244.801004:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202245.825025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202246.849061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202247.873040:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202248.897096:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202249.088996:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202249.089019:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202249.089024:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202249.089036:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1520 00000100:00080000:1.0:1788202249.089050:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202249.089057:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202249.089105:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442853248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202249.089144:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202249.089149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202249.089151:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202249.089156:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1520 00000100:00080000:1.0:1788202249.089160:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202249.089164:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202249.089175:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202249.089191:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442853248:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202249.089192:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442853376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202249.089207:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202249.089210:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202249.089212:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202249.089215:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1520 00000100:00080000:1.0:1788202249.089218:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202249.089221:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202249.089225:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202249.089234:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442853504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202249.089241:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202249.089243:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202249.089244:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202249.089246:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1520 00000100:00080000:1.0:1788202249.089249:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202249.089251:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202249.089254:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202249.089262:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442853632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202249.089263:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202249.089313:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202249/real 1788202249] req@ffff90234519ea00 x1875064442853248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202249.089327:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442853248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202249.089335:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202249.089351:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202249.089355:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442853248:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202249.089377:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442853376:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202249.089395:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202249.089403:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202249/real 1788202249] req@ffff90234519c700 x1875064442853376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202249.089411:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442853376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202249.089417:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202249.089419:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202249.089421:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442853376:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202249.089424:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442853504:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202249.089434:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202249.089439:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202249/real 1788202249] req@ffff90234519fb80 x1875064442853504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202249.089446:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442853504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202249.089451:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202249.089452:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202249.089454:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442853504:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202249.089457:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442853632:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202249.089467:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202249.089472:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202249/real 1788202249] req@ffff90234519df80 x1875064442853632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202249.089483:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442853632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202249.089487:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202249.089489:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202249.089491:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442853632:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202249.921045:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202250.945076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202251.969014:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202252.993024:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202254.017021:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202254.208980:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202254.208987:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202254.208990:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202254.208998:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1525 00000100:00080000:2.0:1788202254.209007:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202254.209012:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202254.209042:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064442853760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202254.209062:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202254.209064:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202254.209066:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202254.209069:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1525 00000100:00080000:2.0:1788202254.209071:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202254.209074:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202254.209082:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202254.209088:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442853760:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202254.209092:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064442853888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202254.209100:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202254.209102:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202254.209103:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202254.209106:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1525 00000100:00080000:2.0:1788202254.209107:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202254.209109:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202254.209112:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202254.209125:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442854016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202254.209133:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202254.209135:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202254.209136:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202254.209139:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1525 00000100:00080000:2.0:1788202254.209141:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202254.209143:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202254.209145:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202254.209147:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202254.209155:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442854144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202254.209180:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202254/real 1788202254] req@ffff90234598ca80 x1875064442853760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202254.209192:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064442853760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202254.209199:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202254.209201:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202254.209204:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442853760:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202254.209220:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442853888:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202254.209235:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202254.209245:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202254/real 1788202254] req@ffff90234598e680 x1875064442853888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202254.209285:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064442853888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202254.209292:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202254.209294:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202254.209296:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442853888:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202254.209300:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442854016:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202254.209314:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202254.209320:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202254/real 1788202254] req@ffff90234598d880 x1875064442854016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202254.209327:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442854016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202254.209332:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202254.209334:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202254.209335:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442854016:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202254.209338:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442854144:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202254.209346:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202254.209351:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202254/real 1788202254] req@ffff90234598f800 x1875064442854144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202254.209357:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442854144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202254.209362:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202254.209363:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202254.209365:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442854144:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202255.041061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202256.064989:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202257.089047:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202258.113007:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202259.137112:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202259.328985:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202259.328996:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202259.329001:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202259.329011:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1530 00000100:00080000:2.0:1788202259.329023:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202259.329031:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202259.329082:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442854272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202259.329113:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202259.329116:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202259.329117:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202259.329121:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1530 00000100:00080000:2.0:1788202259.329124:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202259.329127:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202259.329137:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202259.329150:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442854400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202259.329175:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442854272:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202259.329179:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202259.329184:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202259.329186:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202259.329190:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1530 00000100:00080000:2.0:1788202259.329192:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202259.329195:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202259.329200:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202259.329212:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442854528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202259.329225:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202259.329227:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202259.329228:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202259.329231:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1530 00000100:00080000:2.0:1788202259.329236:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202259.329238:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202259.329241:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202259.329248:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202259.329251:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442854656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202259.329293:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202259/real 1788202259] req@ffff90234598dc00 x1875064442854272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202259.329312:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442854272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202259.329323:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202259.329328:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202259.329332:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442854272:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202259.329360:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442854400:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202259.329380:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202259.329390:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202259/real 1788202259] req@ffff90234598c700 x1875064442854400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202259.329404:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442854400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202259.329410:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202259.329412:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202259.329414:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442854400:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202259.329418:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442854528:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202259.329430:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202259.329437:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202259/real 1788202259] req@ffff90234598ce00 x1875064442854528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202259.329444:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442854528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202259.329450:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202259.329451:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202259.329453:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442854528:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202259.329456:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442854656:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202259.329466:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202259.329472:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202259/real 1788202259] req@ffff90234598e300 x1875064442854656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202259.329479:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442854656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202259.329489:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202259.329491:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202259.329493:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442854656:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202260.161034:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202261.185034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202262.209018:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202263.233048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202264.257148:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202264.448990:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202264.449002:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202264.449007:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202264.449018:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1535 00000100:00080000:2.0:1788202264.449030:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202264.449038:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202264.449091:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442854784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202264.449119:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202264.449124:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202264.449127:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202264.449133:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1535 00000100:00080000:2.0:1788202264.449138:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202264.449144:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202264.449153:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442854784:192.168.201.130@tcp:250:kworker.0 00000008:00100000:2.0:1788202264.449155:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202264.449177:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442854912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202264.449189:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202264.449193:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202264.449196:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202264.449201:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1535 00000100:00080000:2.0:1788202264.449206:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202264.449212:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:1.0:1788202264.449217:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:2.0:1788202264.449219:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202264.449236:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442855040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202264.449248:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202264.449252:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202264.449255:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202264.449261:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1535 00000100:00100000:1.0:1788202264.449262:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202264/real 1788202264] req@ffff90234598f480 x1875064442854784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202264.449268:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202264.449276:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202264.449284:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202264.449296:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d500 x1875064442855168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202264.449305:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442854784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202264.449333:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202264.449338:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202264.449344:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442854784:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202264.449370:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442854912:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202264.449408:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202264.449423:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202264/real 1788202264] req@ffff90234598c380 x1875064442854912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202264.449437:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442854912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202264.449454:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202264.449458:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202264.449462:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442854912:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202264.449468:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442855040:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202264.449486:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202264.449495:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202264/real 1788202264] req@ffff90234598d180 x1875064442855040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202264.449505:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442855040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202264.449511:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202264.449514:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202264.449516:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442855040:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202264.449520:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442855168:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202264.449533:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202264.449541:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202264/real 1788202264] req@ffff90234598d500 x1875064442855168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202264.449549:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d500 x1875064442855168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202264.449556:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202264.449560:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202264.449565:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442855168:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202265.281026:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202266.305036:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202267.329075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202268.353105:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202269.377018:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202269.568977:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202269.569003:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202269.569007:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202269.569018:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1541 00000100:00080000:1.0:1788202269.569030:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202269.569037:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202269.569080:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442855296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202269.569107:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202269.569110:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202269.569112:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202269.569115:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1541 00000100:00080000:1.0:1788202269.569118:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202269.569122:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202269.569134:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202269.569149:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442855424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202269.569154:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442855296:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202269.569172:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202269.569176:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202269.569178:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202269.569182:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1541 00000100:00080000:1.0:1788202269.569186:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202269.569189:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202269.569194:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202269.569206:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442855552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202269.569213:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202269.569216:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202269.569217:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202269.569220:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1541 00000100:00080000:1.0:1788202269.569222:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202269.569225:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202269.569228:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202269.569239:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442855680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202269.569244:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202269.569290:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202269/real 1788202269] req@ffff90234519ed80 x1875064442855296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202269.569307:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442855296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202269.569316:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202269.569320:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202269.569325:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442855296:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202269.569355:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442855424:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202269.569388:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202269.569399:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202269/real 1788202269] req@ffff90234519d880 x1875064442855424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202269.569408:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442855424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202269.569414:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202269.569416:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202269.569419:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442855424:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202269.569426:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442855552:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202269.569440:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202269.569447:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202269/real 1788202269] req@ffff90234519c000 x1875064442855552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202269.569458:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442855552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202269.569467:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202269.569471:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202269.569475:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442855552:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202269.569482:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442855680:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202269.569503:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202269.569516:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202269/real 1788202269] req@ffff90234519e300 x1875064442855680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202269.569529:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442855680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202269.569539:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202269.569541:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202269.569544:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442855680:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202270.401116:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202271.425035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202272.449106:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202273.473077:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202274.497105:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202274.689050:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202274.689065:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202274.689071:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202274.689082:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1546 00000100:00080000:3.0:1788202274.689107:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202274.689117:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202274.689182:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064442855808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202274.689216:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202274.689220:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202274.689222:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202274.689228:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1546 00000100:00080000:3.0:1788202274.689233:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202274.689239:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202274.689252:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202274.689268:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442855808:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202274.689272:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442855936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202274.689286:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202274.689291:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202274.689292:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202274.689296:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1546 00000100:00080000:3.0:1788202274.689300:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202274.689303:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202274.689307:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202274.689321:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442856064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202274.689331:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202274.689335:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202274.689338:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202274.689343:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1546 00000100:00080000:3.0:1788202274.689347:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202274.689355:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:2.0:1788202274.689356:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:3.0:1788202274.689365:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202274.689381:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442856192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202274.689410:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202274/real 1788202274] req@ffff90235268aa00 x1875064442855808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202274.689428:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064442855808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202274.689437:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202274.689440:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202274.689444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442855808:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202274.689476:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442855936:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202274.689503:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202274.689518:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202274/real 1788202274] req@ffff902352688a80 x1875064442855936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202274.689533:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442855936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202274.689542:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202274.689545:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202274.689549:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442855936:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202274.689557:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442856064:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202274.689578:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202274.689589:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202274/real 1788202274] req@ffff90235268b480 x1875064442856064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202274.689610:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442856064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202274.689619:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202274.689622:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202274.689626:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442856064:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202274.689631:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442856192:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202274.689646:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202274.689655:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202274/real 1788202274] req@ffff90235268ad80 x1875064442856192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202274.689667:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442856192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202274.689676:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202274.689679:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202274.689682:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442856192:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202275.521013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202276.545082:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202277.569079:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202278.593063:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202279.617031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202279.808991:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202279.809012:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202279.809017:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202279.809027:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1551 00000100:00080000:2.0:1788202279.809042:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202279.809051:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202279.809110:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442856320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202279.809151:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202279.809156:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202279.809158:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202279.809163:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1551 00000100:00080000:2.0:1788202279.809168:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202279.809174:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202279.809187:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202279.809205:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442856320:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202279.809206:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442856448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202279.809221:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202279.809224:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202279.809226:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202279.809231:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1551 00000100:00080000:2.0:1788202279.809236:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202279.809241:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202279.809247:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202279.809263:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442856576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202279.809274:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202279.809278:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202279.809280:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788202279.809286:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788202279.809291:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1551 00000100:00080000:2.0:1788202279.809302:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202279.809307:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202279.809312:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202279.809326:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442856704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202279.809353:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202279/real 1788202279] req@ffff90234598df80 x1875064442856320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202279.809384:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442856320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202279.809395:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202279.809400:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202279.809405:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442856320:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202279.809436:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442856448:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202279.809460:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202279.809472:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202279/real 1788202279] req@ffff90234598f100 x1875064442856448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202279.809481:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442856448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202279.809487:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202279.809490:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202279.809492:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442856448:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202279.809498:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442856576:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202279.809513:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202279.809520:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202279/real 1788202279] req@ffff90234598ed80 x1875064442856576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202279.809529:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442856576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202279.809535:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202279.809538:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202279.809540:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442856576:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202279.809546:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442856704:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202279.809560:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202279.809566:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202279/real 1788202279] req@ffff90234598fb80 x1875064442856704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202279.809573:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442856704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202279.809579:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202279.809580:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202279.809582:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442856704:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202280.641078:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202281.665112:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202282.689067:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202283.713054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202284.737051:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202284.929020:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202284.929038:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202284.929043:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202284.929055:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1556 00000100:00080000:2.0:1788202284.929068:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202284.929077:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202284.929129:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442856832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202284.929152:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202284.929158:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202284.929160:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202284.929163:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1556 00000100:00080000:2.0:1788202284.929165:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202284.929169:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202284.929178:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202284.929191:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442856960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202284.929205:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202284.929208:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202284.929210:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202284.929213:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1556 00000100:00080000:2.0:1788202284.929215:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202284.929218:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202284.929221:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202284.929229:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442856832:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202284.929237:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442857088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202284.929256:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202284.929261:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202284.929263:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202284.929278:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1556 00000100:00080000:2.0:1788202284.929289:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202284.929295:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202284.929300:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202284.929317:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442857216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202284.929329:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202284.929384:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202284/real 1788202284] req@ffff90234598fb80 x1875064442856832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202284.929405:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442856832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202284.929417:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202284.929422:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202284.929426:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442856832:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202284.929435:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442856960:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202284.929469:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202284.929483:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202284/real 1788202284] req@ffff90234598ed80 x1875064442856960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202284.929497:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442856960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202284.929505:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202284.929509:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202284.929513:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442856960:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202284.929538:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442857088:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202284.929558:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202284.929570:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202284/real 1788202284] req@ffff90234598f100 x1875064442857088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202284.929583:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442857088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202284.929591:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202284.929594:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202284.929598:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442857088:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202284.929614:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442857216:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202284.929634:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202284.929646:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202284/real 1788202284] req@ffff90234598df80 x1875064442857216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202284.929664:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442857216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202284.929673:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202284.929676:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202284.929680:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442857216:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202285.761211:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202286.785157:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202287.809007:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202288.833044:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202289.857035:0:31830:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/3 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1788202289.857068:0:13084:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788202289.857078:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 2/0,0 mode: PW/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788202289.857084:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788202289.857151:0:13084:0:(vvp_io.c:1826:vvp_io_init()) [0x200000bd1:0x42e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788202289.857162:0:13084:0:(vvp_io.c:304:vvp_io_fini()) [0x200000bd1:0x42e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788202289.857189:0:13084:0:(vvp_object.c:117:vvp_conf_set()) [0x200000bd1:0x42e:0x0]: losing layout lock 00010000:00010000:2.0:1788202289.857223:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c59a8 lrc: 2/0,0 mode: PR/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d698b expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788202289.857236:0:13084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c59a1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9383f985bc4d696f expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788202289.857268:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 1/0,0 mode: --/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788202289.857274:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c59a8 lrc: 1/0,0 mode: --/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d698b expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788202289.857278:0:13084:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c59a1 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d696f expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788202289.857282:0:13084:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:2.0:1788202289.857305:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ca00/0x4605c7c97e1c59af lrc: 0/0,0 mode: --/PW res: [0x200000bd1:0x42e:0x0].0x0 bits 0x59/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9383f985bc4d69a7 expref: -99 pid: 31719 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788202289.857318:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265ce00/0x4605c7c97e1c59a8 lrc: 0/0,0 mode: --/PR res: [0x200000bd1:0x42c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d698b expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788202289.857326:0:13084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff902350a11000 lock: ffff90235265c200/0x4605c7c97e1c59a1 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9383f985bc4d696f expref: -99 pid: 31719 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788202289.857340:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202289.857342:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202289.857345:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202289.857350:0:2362:0:(client.c:1832:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff90234598d500 x1875064442857344/t0(0) o103->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:17/18 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202290.049000:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202290.049009:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202290.049013:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202290.049022:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1561 00000100:00080000:1.0:1788202290.049031:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202290.049036:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202290.049071:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442857472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202290.049097:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202290.049100:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202290.049101:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202290.049104:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1561 00000100:00080000:1.0:1788202290.049106:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202290.049109:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202290.049116:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202290.049128:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442857600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202290.049136:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202290.049139:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202290.049142:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202290.049144:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442857472:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202290.049147:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1561 00000100:00080000:1.0:1788202290.049152:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202290.049157:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202290.049160:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202290.049168:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442857728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202290.049175:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202290.049178:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202290.049180:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202290.049182:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1561 00000100:00080000:1.0:1788202290.049185:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202290.049187:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202290.049190:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202290.049196:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442857856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202290.049214:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202290.049250:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202290/real 1788202290] req@ffff90234519f800 x1875064442857472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202290.049264:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442857472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202290.049271:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202290.049274:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202290.049277:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442857472:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202290.049295:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442857600:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202290.049310:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202290.049318:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202290/real 1788202290] req@ffff90234519e300 x1875064442857600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202290.049326:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442857600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202290.049331:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202290.049333:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202290.049335:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442857600:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202290.049341:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442857728:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202290.049352:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202290.049358:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202290/real 1788202290] req@ffff90234519c000 x1875064442857728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202290.049365:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442857728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202290.049370:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202290.049372:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202290.049374:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442857728:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202290.049377:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442857856:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202290.049386:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202290.049391:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202290/real 1788202290] req@ffff90234519d880 x1875064442857856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202290.049398:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442857856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202290.049403:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202290.049405:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202290.049407:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442857856:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202290.881018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202291.905028:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202292.929056:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202293.953050:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202294.977049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202295.169014:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202295.169029:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202295.169033:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202295.169044:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1566 00000100:00080000:2.0:1788202295.169055:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202295.169062:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202295.169100:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442857984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202295.169131:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202295.169135:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202295.169138:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202295.169143:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1566 00000100:00080000:2.0:1788202295.169148:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202295.169153:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202295.169162:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442857984:192.168.201.130@tcp:250:kworker.0 00000008:00100000:2.0:1788202295.169163:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202295.169182:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442858112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202295.169194:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202295.169198:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202295.169200:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202295.169204:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1566 00000100:00080000:2.0:1788202295.169209:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202295.169213:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202295.169219:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202295.169224:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202295.169233:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442858240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202295.169245:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202295.169249:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202295.169251:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202295.169256:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1566 00000100:00080000:2.0:1788202295.169260:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202295.169265:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202295/real 1788202295] req@ffff90234598d180 x1875064442857984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202295.169268:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202295.169276:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202295.169292:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442858368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202295.169298:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442857984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202295.169315:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202295.169319:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202295.169322:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442857984:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202295.169343:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442858112:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202295.169362:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202295.169371:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202295/real 1788202295] req@ffff90234598c380 x1875064442858112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202295.169380:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442858112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202295.169385:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202295.169388:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202295.169390:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442858112:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202295.169394:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442858240:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202295.169404:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202295.169411:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202295/real 1788202295] req@ffff90234598f480 x1875064442858240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202295.169430:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442858240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202295.169439:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202295.169442:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202295.169446:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442858240:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202295.169452:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442858368:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202295.169471:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202295.169480:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202295/real 1788202295] req@ffff90234598e300 x1875064442858368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202295.169488:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442858368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202295.169493:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202295.169495:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202295.169497:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442858368:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202296.001109:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202297.025105:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202298.049117:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202299.073044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202300.097044:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202300.289115:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202300.289123:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202300.289127:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202300.289139:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1571 00000100:00080000:2.0:1788202300.289152:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202300.289160:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202300.289203:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442858496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202300.289233:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202300.289237:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202300.289239:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202300.289244:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1571 00000100:00080000:2.0:1788202300.289248:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202300.289253:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202300.289264:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202300.289277:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442858624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202300.289285:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202300.289288:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202300.289290:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202300.289296:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1571 00000100:00080000:2.0:1788202300.289302:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202300.289306:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202300.289316:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202300.289317:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442858496:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202300.289332:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442858752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202300.289340:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202300.289344:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202300.289345:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202300.289348:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1571 00000100:00080000:2.0:1788202300.289349:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202300.289353:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202300.289364:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202300.289373:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442858880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202300.289417:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202300.289516:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202300/real 1788202300] req@ffff90234598ce00 x1875064442858496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202300.289554:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442858496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202300.289570:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202300.289577:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202300.289583:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442858496:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202300.289595:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442858624:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202300.289638:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202300.289665:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202300/real 1788202300] req@ffff90234598c700 x1875064442858624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202300.289702:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442858624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202300.289714:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202300.289720:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202300.289724:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442858624:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202300.289761:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442858752:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202300.289801:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202300.289825:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202300/real 1788202300] req@ffff90234598dc00 x1875064442858752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202300.289843:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442858752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202300.289860:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202300.289864:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202300.289868:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442858752:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202300.289932:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442858880:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202300.289970:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202300.289990:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202300/real 1788202300] req@ffff90234598f800 x1875064442858880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202300.290008:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442858880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202300.290018:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202300.290022:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202300.290026:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442858880:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202301.121037:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202302.145033:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202303.169045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202304.193066:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202305.217042:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202305.408992:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202305.409001:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202305.409005:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202305.409013:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1576 00000100:00080000:3.0:1788202305.409022:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202305.409028:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202305.409077:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442859008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202305.409106:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202305.409108:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202305.409110:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202305.409113:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1576 00000100:00080000:3.0:1788202305.409116:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202305.409119:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202305.409137:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202305.409149:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442859136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202305.409156:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202305.409159:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202305.409160:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202305.409163:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1576 00000100:00080000:3.0:1788202305.409165:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202305.409167:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202305.409171:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202305.409181:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442859008:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202305.409181:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442859264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202305.409207:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202305.409213:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202305.409216:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202305.409223:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1576 00000100:00080000:3.0:1788202305.409227:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202305.409233:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202305.409239:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202305.409250:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442859392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202305.409340:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442859136:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202305.409427:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442859264:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202305.409458:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442859392:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202306.241013:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202307.265059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202308.288981:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788202308.481106:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:0.0:1788202308.481145:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788202308.481161:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788202308.481206:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788202308.481218:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788202308.481226:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:1.0:1788202308.481380:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202305/real 1788202308] req@ffff902352688380 x1875064442859264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202308.481409:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442859264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202308.481423:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202308.481438:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202308.481442:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442859264:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202308.481451:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202305/real 1788202308] req@ffff902352689880 x1875064442859392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202308.481465:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442859392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202308.481473:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202308.481477:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202308.481481:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442859392:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202308.481488:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202305/real 1788202308] req@ffff90235268bb80 x1875064442859008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202308.481500:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442859008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202308.481508:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202308.481511:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202308.481514:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442859008:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202308.481521:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202305/real 1788202308] req@ffff902352688700 x1875064442859136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202308.481534:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442859136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202308.481543:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202308.481546:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202308.481549:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442859136:192.168.201.130@tcp:38:kworker.0 00010000:00010000:2.0:1788202309.313013:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202310.336992:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202310.529009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202310.529019:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202310.529025:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202310.529036:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1582 00000100:00080000:3.0:1788202310.529049:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202310.529058:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202310.529095:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442859520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202310.529118:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202310.529121:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202310.529123:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202310.529126:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1582 00000100:00080000:3.0:1788202310.529129:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202310.529132:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202310.529139:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202310.529147:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442859648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202310.529155:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202310.529158:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202310.529159:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202310.529161:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1582 00000100:00080000:3.0:1788202310.529163:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202310.529165:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202310.529168:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202310.529173:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442859776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202310.529181:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202310.529185:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202310.529187:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202310.529197:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1582 00000100:00100000:1.0:1788202310.529202:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442859520:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202310.529203:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202310.529213:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202310.529219:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202310.529232:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442859904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202310.529313:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202310.529359:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202310/real 1788202310] req@ffff902352689180 x1875064442859520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202310.529397:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442859520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202310.529409:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202310.529413:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202310.529424:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442859520:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202310.529433:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442859648:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202310.529459:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202310.529474:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202310/real 1788202310] req@ffff902352689500 x1875064442859648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202310.529489:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442859648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202310.529503:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202310.529507:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202310.529511:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442859648:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202310.529517:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442859776:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202310.529538:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202310.529550:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202310/real 1788202310] req@ffff90235268a680 x1875064442859776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202310.529569:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442859776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202310.529579:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202310.529583:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202310.529587:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442859776:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202310.529619:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442859904:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202310.529641:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202310.529652:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202310/real 1788202310] req@ffff902352689c00 x1875064442859904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202310.529663:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442859904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202310.529671:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202310.529674:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202310.529677:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442859904:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202311.361001:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202312.385020:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202313.409200:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202314.433028:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202315.457083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202315.649010:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202315.649021:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202315.649025:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202315.649036:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1587 00000100:00080000:1.0:1788202315.649049:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202315.649056:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202315.649111:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442860032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202315.649140:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202315.649145:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202315.649147:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202315.649152:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1587 00000100:00080000:1.0:1788202315.649156:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202315.649163:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202315.649174:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202315.649201:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442860160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202315.649212:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788202315.649212:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442860032:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202315.649216:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202315.649218:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202315.649222:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1587 00000100:00080000:1.0:1788202315.649226:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202315.649230:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202315.649235:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202315.649245:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442860288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202315.649254:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202315.649257:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202315.649259:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202315.649262:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1587 00000100:00080000:1.0:1788202315.649265:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202315.649279:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202315.649283:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788202315.649294:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202315.649295:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442860416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202315.649353:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202315/real 1788202315] req@ffff90234519ed80 x1875064442860032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202315.649376:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442860032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202315.649389:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202315.649393:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202315.649399:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442860032:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202315.649427:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442860160:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202315.649448:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202315.649458:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202315/real 1788202315] req@ffff90234519df80 x1875064442860160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202315.649467:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442860160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202315.649478:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202315.649480:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202315.649483:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442860160:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202315.649487:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442860288:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202315.649500:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202315.649510:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202315/real 1788202315] req@ffff90234519fb80 x1875064442860288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202315.649522:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442860288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202315.649532:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202315.649535:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202315.649539:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442860288:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202315.649546:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442860416:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202315.649565:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202315.649573:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202315/real 1788202315] req@ffff90234519c700 x1875064442860416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202315.649581:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442860416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202315.649587:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202315.649589:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202315.649592:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442860416:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202316.481073:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202317.505044:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202318.529129:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202319.553034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202320.577012:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202320.769038:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202320.769047:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202320.769050:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202320.769059:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1592 00000100:00080000:3.0:1788202320.769069:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202320.769074:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202320.769132:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442860544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202320.769155:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202320.769158:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202320.769159:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202320.769162:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1592 00000100:00080000:3.0:1788202320.769170:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202320.769174:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202320.769181:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202320.769191:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442860672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202320.769200:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202320.769202:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202320.769204:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202320.769207:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1592 00000100:00080000:3.0:1788202320.769209:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788202320.769213:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442860544:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202320.769214:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202320.769220:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202320.769229:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442860800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202320.769236:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202320.769239:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202320.769240:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202320.769242:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1592 00000100:00080000:3.0:1788202320.769245:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202320.769247:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202320.769249:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202320.769256:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442860928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202320.769295:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202320.769336:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202320/real 1788202320] req@ffff902352688e00 x1875064442860544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202320.769355:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442860544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202320.769366:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202320.769371:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202320.769375:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442860544:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202320.769411:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442860672:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202320.769434:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202320.769447:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202320/real 1788202320] req@ffff90235268a300 x1875064442860672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202320.769459:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442860672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202320.769467:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202320.769470:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202320.769473:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442860672:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202320.769480:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442860800:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202320.769498:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202320.769508:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202320/real 1788202320] req@ffff902352688000 x1875064442860800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202320.769520:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442860800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202320.769528:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202320.769532:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202320.769535:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442860800:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202320.769541:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442860928:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202320.769557:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202320.769567:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202320/real 1788202320] req@ffff90235268b800 x1875064442860928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202320.769579:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442860928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202320.769586:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202320.769589:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202320.769593:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442860928:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202321.600995:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202322.625079:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202323.649062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202324.673058:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202325.697085:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202325.889037:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202325.889047:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202325.889051:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202325.889062:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1597 00000100:00080000:2.0:1788202325.889075:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202325.889083:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202325.889127:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442861056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202325.889156:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202325.889160:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202325.889161:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202325.889164:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1597 00000100:00080000:2.0:1788202325.889167:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202325.889169:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202325.889179:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202325.889193:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064442861184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202325.889205:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788202325.889211:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442861056:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202325.889212:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202325.889217:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202325.889224:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1597 00000100:00080000:2.0:1788202325.889229:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202325.889234:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202325.889240:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202325.889253:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064442861312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202325.889261:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202325.889264:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202325.889266:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202325.889269:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1597 00000100:00080000:2.0:1788202325.889273:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202325.889275:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202325.889279:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202325.889288:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442861440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202325.889303:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202325.889368:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202325/real 1788202325] req@ffff90234598d880 x1875064442861056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202325.889392:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442861056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202325.889412:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202325.889416:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202325.889420:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442861056:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202325.889445:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442861184:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202325.889466:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202325.889476:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202325/real 1788202325] req@ffff90234598e680 x1875064442861184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202325.889485:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064442861184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202325.889491:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202325.889493:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202325.889495:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442861184:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202325.889499:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442861312:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202325.889511:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202325.889517:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202325/real 1788202325] req@ffff90234598ca80 x1875064442861312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202325.889525:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064442861312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202325.889530:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202325.889532:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202325.889534:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442861312:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202325.889538:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442861440:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202325.889548:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202325.889554:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202325/real 1788202325] req@ffff90234598f800 x1875064442861440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202325.889561:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442861440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202325.889567:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202325.889568:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202325.889570:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442861440:192.168.201.130@tcp:8:kworker.0 00010000:00010000:3.0:1788202326.721089:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202327.745032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202328.769066:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202329.793053:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202330.817014:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202331.009019:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202331.009030:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202331.009034:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202331.009044:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1602 00000100:00080000:3.0:1788202331.009055:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202331.009062:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202331.009098:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442861568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202331.009125:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202331.009129:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202331.009131:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202331.009136:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1602 00000100:00080000:3.0:1788202331.009140:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202331.009145:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202331.009155:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202331.009172:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442861696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202331.009179:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442861568:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202331.009199:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202331.009203:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202331.009206:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202331.009210:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1602 00000100:00080000:3.0:1788202331.009214:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202331.009218:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202331.009224:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202331.009237:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442861824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202331.009249:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202331.009263:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202331.009265:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202331.009270:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1602 00000100:00080000:3.0:1788202331.009274:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202331.009278:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202331.009285:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202331.009297:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442861952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202331.009304:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202331.009351:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202331/real 1788202331] req@ffff90235268b100 x1875064442861568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202331.009372:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442861568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202331.009384:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202331.009388:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202331.009392:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442861568:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202331.009415:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442861696:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202331.009439:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202331.009454:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202331/real 1788202331] req@ffff90235268b800 x1875064442861696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202331.009467:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442861696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202331.009476:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202331.009479:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202331.009483:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442861696:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202331.009489:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442861824:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202331.009510:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202331.009521:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202331/real 1788202331] req@ffff902352688000 x1875064442861824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202331.009533:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442861824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202331.009542:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202331.009546:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202331.009549:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442861824:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202331.009555:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442861952:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202331.009573:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202331.009583:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202331/real 1788202331] req@ffff90235268a300 x1875064442861952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202331.009595:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442861952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202331.009604:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202331.009607:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202331.009610:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442861952:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202331.841020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202332.865057:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202333.889003:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202334.912997:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202335.937039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202336.128991:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202336.129002:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202336.129006:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202336.129018:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1607 00000100:00080000:3.0:1788202336.129029:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202336.129038:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202336.129078:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442862080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202336.129108:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202336.129117:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202336.129120:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202336.129124:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1607 00000100:00080000:3.0:1788202336.129129:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202336.129133:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202336.129144:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202336.129163:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442862208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202336.129176:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442862080:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202336.129178:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202336.129188:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202336.129191:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202336.129197:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1607 00000100:00080000:3.0:1788202336.129202:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202336.129207:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202336.129213:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202336.129228:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442862336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202336.129240:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202336.129244:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202336.129246:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788202336.129250:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788202336.129256:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1607 00000100:00080000:3.0:1788202336.129262:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202336.129296:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202336/real 1788202336] req@ffff902352688e00 x1875064442862080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202336.129320:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442862080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202336.129332:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202336.129337:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00080000:1.0:1788202336.129338:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202336.129346:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442862080:192.168.201.130@tcp:250:kworker.0 00000008:00100000:3.0:1788202336.129358:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202336.129370:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442862208:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202336.129380:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442862464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202336.129396:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202336.129411:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202336/real 1788202336] req@ffff902352689c00 x1875064442862208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202336.129425:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442862208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202336.129436:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202336.129440:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202336.129443:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442862208:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202336.129450:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442862336:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202336.129473:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202336.129482:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202336/real 1788202336] req@ffff90235268a680 x1875064442862336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202336.129491:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442862336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202336.129497:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202336.129500:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202336.129502:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442862336:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202336.129513:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442862464:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202336.129527:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202336.129535:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202336/real 1788202336] req@ffff902352689500 x1875064442862464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202336.129555:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442862464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202336.129560:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202336.129564:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202336.129567:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442862464:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202336.961036:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202337.985040:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202339.008991:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202340.033031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202341.057051:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202341.248982:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202341.248992:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202341.248996:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202341.249007:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1612 00000100:00080000:3.0:1788202341.249017:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202341.249023:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202341.249065:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442862592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202341.249091:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202341.249095:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202341.249097:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202341.249101:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1612 00000100:00080000:3.0:1788202341.249104:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202341.249108:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202341.249118:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202341.249130:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442862720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202341.249142:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788202341.249147:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442862592:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202341.249148:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202341.249152:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202341.249158:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1612 00000100:00080000:3.0:1788202341.249161:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202341.249165:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202341.249170:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202341.249183:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442862848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202341.249193:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202341.249196:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202341.249198:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202341.249202:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1612 00000100:00080000:3.0:1788202341.249206:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202341.249210:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:1.0:1788202341.249216:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:3.0:1788202341.249216:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202341.249230:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442862976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202341.249258:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202341/real 1788202341] req@ffff902352689180 x1875064442862592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202341.249276:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442862592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202341.249287:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202341.249291:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202341.249295:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442862592:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202341.249318:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442862720:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202341.249342:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202341.249357:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202341/real 1788202341] req@ffff902352688700 x1875064442862720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202341.249370:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442862720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202341.249380:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202341.249383:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202341.249387:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442862720:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202341.249393:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442862848:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202341.249415:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202341.249426:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202341/real 1788202341] req@ffff90235268bb80 x1875064442862848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202341.249446:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442862848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202341.249455:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202341.249458:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202341.249461:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442862848:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202341.249467:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442862976:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202341.249483:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202341.249492:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202341/real 1788202341] req@ffff902352689880 x1875064442862976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202341.249504:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442862976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202341.249512:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202341.249514:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202341.249517:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442862976:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202342.081106:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202343.105020:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202344.129056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202345.153028:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202346.177127:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202346.369068:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202346.369079:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202346.369083:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202346.369094:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1617 00000100:00080000:2.0:1788202346.369106:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202346.369114:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202346.369154:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442863104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202346.369187:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202346.369191:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202346.369193:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202346.369197:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1617 00000100:00080000:2.0:1788202346.369201:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202346.369205:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202346.369215:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202346.369228:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442863232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202346.369240:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202346.369244:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202346.369246:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202346.369250:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1617 00000100:00080000:2.0:1788202346.369254:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202346.369259:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202346.369265:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442863104:192.168.201.130@tcp:250:kworker.0 00000008:00100000:2.0:1788202346.369267:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202346.369282:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442863360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202346.369294:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202346.369298:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202346.369300:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202346.369305:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1617 00000100:00080000:2.0:1788202346.369309:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202346.369313:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202346.369319:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202346.369325:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202346.369334:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442863488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202346.369377:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202346/real 1788202346] req@ffff90234598dc00 x1875064442863104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202346.369399:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442863104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202346.369410:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202346.369414:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202346.369419:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442863104:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202346.369426:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442863232:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202346.369449:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202346.369461:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202346/real 1788202346] req@ffff90234598c700 x1875064442863232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202346.369474:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442863232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202346.369481:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202346.369483:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202346.369485:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442863232:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202346.369509:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442863360:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202346.369530:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202346.369554:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202346/real 1788202346] req@ffff90234598ce00 x1875064442863360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202346.369569:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442863360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202346.369576:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202346.369579:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202346.369581:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442863360:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202346.369585:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442863488:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202346.369600:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202346.369607:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202346/real 1788202346] req@ffff90234598e300 x1875064442863488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202346.369615:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442863488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202346.369627:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202346.369629:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202346.369631:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442863488:192.168.201.130@tcp:8:kworker.0 00010000:00010000:3.0:1788202347.201026:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202348.225103:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202349.249047:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202350.273086:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202351.296982:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202351.489027:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202351.489043:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202351.489048:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202351.489062:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1623 00000100:00080000:2.0:1788202351.489077:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202351.489087:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202351.489139:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442863616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202351.489170:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202351.489175:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202351.489178:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202351.489183:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1623 00000100:00080000:2.0:1788202351.489188:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202351.489193:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202351.489206:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202351.489220:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442863616:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202351.489225:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442863744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202351.489241:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202351.489245:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202351.489247:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202351.489252:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1623 00000100:00080000:2.0:1788202351.489256:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202351.489260:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202351.489267:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202351.489281:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442863872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202351.489289:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202351.489292:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202351.489295:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788202351.489296:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788202351.489301:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1623 00000100:00080000:2.0:1788202351.489305:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202351.489308:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202351.489312:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202351.489332:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442864000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202351.489353:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202351/real 1788202351] req@ffff90234598f480 x1875064442863616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202351.489370:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442863616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202351.489379:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202351.489383:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202351.489386:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442863616:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202351.489409:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442863744:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202351.489426:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202351.489435:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202351/real 1788202351] req@ffff90234598c380 x1875064442863744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202351.489443:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442863744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202351.489449:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202351.489451:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202351.489453:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442863744:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202351.489457:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442863872:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202351.489468:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202351.489474:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202351/real 1788202351] req@ffff90234598d180 x1875064442863872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202351.489481:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442863872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202351.489487:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202351.489489:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202351.489491:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442863872:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202351.489494:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442864000:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202351.489504:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202351.489509:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202351/real 1788202351] req@ffff90234598df80 x1875064442864000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202351.489525:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442864000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202351.489533:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202351.489535:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202351.489538:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442864000:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202352.321100:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202353.345015:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202354.369065:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202355.393002:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202356.417093:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202356.608998:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202356.609009:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202356.609014:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202356.609024:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1628 00000100:00080000:2.0:1788202356.609036:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202356.609043:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202356.609087:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442864128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202356.609118:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202356.609121:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202356.609127:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202356.609133:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1628 00000100:00080000:2.0:1788202356.609137:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202356.609141:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202356.609155:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202356.609171:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442864256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202356.609184:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788202356.609188:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442864128:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202356.609190:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202356.609195:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202356.609202:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1628 00000100:00080000:2.0:1788202356.609207:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202356.609211:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202356.609218:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202356.609232:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442864384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202356.609246:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202356.609249:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202356.609250:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202356.609253:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1628 00000100:00080000:2.0:1788202356.609264:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202356.609267:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202356.609271:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202356.609272:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202356.609284:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442864512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202356.609324:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202356/real 1788202356] req@ffff90234598f100 x1875064442864128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202356.609342:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442864128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202356.609352:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202356.609356:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202356.609361:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442864128:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202356.609392:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442864256:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202356.609417:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202356.609428:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202356/real 1788202356] req@ffff90234598ed80 x1875064442864256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202356.609437:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442864256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202356.609453:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202356.609456:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202356.609458:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442864256:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202356.609464:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442864384:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202356.609498:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202356.609508:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202356/real 1788202356] req@ffff90234598fb80 x1875064442864384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202356.609516:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442864384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202356.609522:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202356.609525:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202356.609527:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442864384:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202356.609531:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442864512:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202356.609549:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202356.609557:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202356/real 1788202356] req@ffff90234598df80 x1875064442864512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202356.609566:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442864512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202356.609571:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202356.609574:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202356.609576:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442864512:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202357.441063:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202358.465019:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202359.489014:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202360.513030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202361.537084:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202361.729016:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202361.729028:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202361.729033:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202361.729045:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1633 00000100:00080000:2.0:1788202361.729059:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202361.729066:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202361.729112:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442864640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202361.729137:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202361.729140:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202361.729142:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202361.729147:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1633 00000100:00080000:2.0:1788202361.729150:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202361.729152:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202361.729162:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202361.729172:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442864768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202361.729181:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202361.729184:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202361.729186:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202361.729193:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1633 00000100:00100000:1.0:1788202361.729195:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442864640:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202361.729198:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202361.729206:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202361.729212:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202361.729229:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442864896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202361.729241:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202361.729244:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202361.729246:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202361.729249:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1633 00000100:00080000:2.0:1788202361.729252:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202361.729255:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202361.729259:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202361.729269:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442865024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202361.729277:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202361.729328:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202361/real 1788202361] req@ffff90234598d180 x1875064442864640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202361.729345:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442864640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202361.729357:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202361.729361:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202361.729366:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442864640:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202361.729374:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442864768:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202361.729418:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202361.729430:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202361/real 1788202361] req@ffff90234598c380 x1875064442864768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202361.729439:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442864768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202361.729445:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202361.729447:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202361.729450:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442864768:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202361.729475:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442864896:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202361.729489:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202361.729497:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202361/real 1788202361] req@ffff90234598f480 x1875064442864896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202361.729511:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442864896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202361.729520:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202361.729524:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202361.729527:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442864896:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202361.729533:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442865024:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202361.729554:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202361.729566:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202361/real 1788202361] req@ffff90234598e300 x1875064442865024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202361.729578:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442865024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202361.729586:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202361.729589:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202361.729593:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442865024:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202362.561064:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202363.584975:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202364.609068:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202365.633043:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202366.657037:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202366.849093:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202366.849101:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202366.849104:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202366.849113:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1638 00000100:00080000:3.0:1788202366.849122:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202366.849129:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202366.849172:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442865152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202366.849203:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202366.849208:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202366.849210:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202366.849214:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1638 00000100:00080000:3.0:1788202366.849217:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202366.849221:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202366.849230:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202366.849242:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442865280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202366.849256:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442865152:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202366.849256:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202366.849264:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202366.849267:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202366.849271:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1638 00000100:00080000:3.0:1788202366.849273:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202366.849276:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202366.849280:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202366.849288:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442865408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202366.849309:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202366.849312:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202366.849314:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202366.849317:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1638 00000100:00080000:3.0:1788202366.849320:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202366.849324:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202366.849328:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202366.849337:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202366.849340:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442865536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202366.849377:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202366/real 1788202366] req@ffff902352688380 x1875064442865152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202366.849395:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442865152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202366.849403:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202366.849406:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202366.849410:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442865152:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202366.849433:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442865280:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202366.849451:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202366.849460:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202366/real 1788202366] req@ffff90235268ad80 x1875064442865280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202366.849469:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442865280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202366.849475:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202366.849477:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202366.849480:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442865280:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202366.849484:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442865408:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202366.849495:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202366.849501:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202366/real 1788202366] req@ffff90235268b480 x1875064442865408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202366.849509:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442865408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202366.849514:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202366.849516:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202366.849519:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442865408:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202366.849523:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442865536:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202366.849532:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202366.849538:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202366/real 1788202366] req@ffff902352688a80 x1875064442865536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202366.849545:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442865536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202366.849550:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202366.849552:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202366.849553:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442865536:192.168.201.130@tcp:8:kworker.0 00010000:00010000:3.0:1788202367.681039:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202368.705083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202369.729068:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202370.753068:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202371.777005:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202371.968989:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202371.968999:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202371.969004:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202371.969014:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1643 00000100:00080000:0.0:1788202371.969026:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202371.969034:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202371.969069:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442865664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202371.969098:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202371.969102:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202371.969104:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202371.969110:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1643 00000100:00080000:0.0:1788202371.969115:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202371.969121:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202371.969131:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202371.969140:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442865664:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202371.969147:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442865792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202371.969158:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202371.969160:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202371.969162:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202371.969166:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1643 00000100:00080000:0.0:1788202371.969169:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202371.969173:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202371.969177:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202371.969190:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442865920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202371.969200:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202371.969215:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202371.969218:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202371.969223:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1643 00000100:00080000:0.0:1788202371.969228:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202371.969234:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202371.969240:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202371.969255:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442866048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202371.969260:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442865792:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202371.969357:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442865920:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202371.969386:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442866048:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202372.801051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202373.826469:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202374.849031:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788202375.041147:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788202375.041255:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788202375.041273:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202375.041312:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202375.041322:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202375.041330:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788202375.041371:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202371/real 1788202375] req@ffff9023450b9500 x1875064442866048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202375.041408:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442866048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202375.041420:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202375.041426:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202375.041430:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442866048:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202375.041439:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202371/real 1788202375] req@ffff9023450bb800 x1875064442865792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202375.041454:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442865792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202375.041477:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202375.041482:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202375.041486:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442865792:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202375.041494:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202371/real 1788202375] req@ffff9023450b8a80 x1875064442865920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202375.041508:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442865920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202375.041517:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202375.041521:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202375.041524:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442865920:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202375.041542:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202371/real 1788202375] req@ffff9023450b8380 x1875064442865664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202375.041561:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442865664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202375.041569:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202375.041572:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202375.041575:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442865664:192.168.201.130@tcp:250:kworker.0 00010000:00010000:3.0:1788202375.873013:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202376.897278:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202377.088987:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202377.088998:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202377.089001:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202377.089012:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1648 00000100:00080000:3.0:1788202377.089023:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202377.089030:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202377.089061:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064442866176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202377.089084:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202377.089091:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202377.089093:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202377.089096:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1648 00000100:00080000:3.0:1788202377.089099:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202377.089103:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202377.089111:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202377.089132:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442866176:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202377.089143:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442866304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202377.089162:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202377.089166:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202377.089168:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202377.089172:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1648 00000100:00080000:3.0:1788202377.089177:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202377.089182:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202377.089188:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202377.089203:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202377.089204:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442866432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202377.089218:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202377.089223:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202377.089225:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202377.089229:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1648 00000100:00080000:3.0:1788202377.089234:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202377.089239:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202377.089245:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202377.089247:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202377/real 1788202377] req@ffff90235268aa00 x1875064442866176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202377.089259:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442866560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202377.089270:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064442866176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202377.089283:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202377.089287:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202377.089293:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442866176:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202377.089322:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442866304:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202377.089347:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202377.089361:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202377/real 1788202377] req@ffff902352688a80 x1875064442866304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202377.089373:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442866304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202377.089381:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202377.089384:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202377.089388:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442866304:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202377.089394:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442866432:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202377.089413:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202377.089425:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202377/real 1788202377] req@ffff90235268b480 x1875064442866432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202377.089437:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442866432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202377.089445:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202377.089448:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202377.089452:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442866432:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202377.089458:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442866560:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202377.089477:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202377.089487:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202377/real 1788202377] req@ffff90235268ad80 x1875064442866560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202377.089499:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442866560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202377.089507:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202377.089511:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202377.089513:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442866560:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202377.921314:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202378.945123:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202379.969078:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202380.993057:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202382.017071:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202382.208995:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202382.209003:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202382.209006:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202382.209014:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1653 00000100:00080000:2.0:1788202382.209040:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202382.209047:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202382.209085:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442866688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202382.209115:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202382.209118:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202382.209120:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202382.209123:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1653 00000100:00080000:2.0:1788202382.209127:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202382.209131:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202382.209141:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202382.209158:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442866816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202382.209166:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442866688:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202382.209177:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202382.209181:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202382.209183:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202382.209189:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1653 00000100:00080000:2.0:1788202382.209193:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202382.209198:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202382.209204:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202382.209220:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442866944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202382.209231:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788202382.209237:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788202382.209238:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202382.209242:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202382.209248:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1653 00000100:00080000:2.0:1788202382.209253:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202382.209257:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202382.209263:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202382.209278:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442867072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202382.209307:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202382/real 1788202382] req@ffff90234598ce00 x1875064442866688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202382.209326:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442866688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202382.209335:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202382.209338:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202382.209341:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442866688:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202382.209363:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442866816:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202382.209389:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202382.209418:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202382/real 1788202382] req@ffff90234598c700 x1875064442866816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202382.209433:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442866816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202382.209439:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202382.209441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202382.209444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442866816:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202382.209448:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442866944:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202382.209464:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202382.209470:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202382/real 1788202382] req@ffff90234598dc00 x1875064442866944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202382.209478:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442866944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202382.209484:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202382.209486:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202382.209488:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442866944:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202382.209492:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442867072:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202382.209501:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202382.209507:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202382/real 1788202382] req@ffff90234598f800 x1875064442867072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202382.209514:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442867072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202382.209524:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202382.209526:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202382.209527:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442867072:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202383.041260:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202384.065068:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202385.089082:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202386.113068:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202387.137084:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202387.329004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202387.329015:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202387.329020:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202387.329032:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1658 00000100:00080000:2.0:1788202387.329044:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202387.329053:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202387.329103:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064442867200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202387.329133:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202387.329137:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202387.329139:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202387.329144:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1658 00000100:00080000:2.0:1788202387.329149:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202387.329154:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202387.329165:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202387.329176:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442867200:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202387.329185:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064442867328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202387.329198:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202387.329201:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202387.329203:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202387.329208:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1658 00000100:00080000:2.0:1788202387.329213:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202387.329217:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202387.329223:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202387.329238:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442867456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202387.329248:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202387.329252:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202387.329257:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788202387.329258:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788202387.329264:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1658 00000100:00080000:2.0:1788202387.329271:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202387.329276:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202387.329282:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202387.329296:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442867584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202387.329305:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202387/real 1788202387] req@ffff90234598ca80 x1875064442867200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202387.329323:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064442867200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202387.329331:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202387.329335:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202387.329338:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442867200:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202387.329357:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442867328:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202387.329375:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202387.329384:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202387/real 1788202387] req@ffff90234598e680 x1875064442867328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202387.329392:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064442867328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202387.329402:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202387.329404:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202387.329407:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442867328:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202387.329410:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442867456:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202387.329421:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202387.329428:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202387/real 1788202387] req@ffff90234598d880 x1875064442867456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202387.329435:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442867456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202387.329441:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202387.329442:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202387.329444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442867456:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202387.329448:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442867584:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202387.329457:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202387.329463:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202387/real 1788202387] req@ffff90234598f800 x1875064442867584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202387.329470:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442867584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202387.329475:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202387.329477:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202387.329479:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442867584:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202388.160978:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202389.185074:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202390.209025:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202391.233032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202392.256996:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202392.448989:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202392.448999:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202392.449003:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202392.449013:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1663 00000100:00080000:2.0:1788202392.449026:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202392.449034:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202392.449073:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442867712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202392.449104:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202392.449109:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202392.449111:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202392.449116:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1663 00000100:00080000:2.0:1788202392.449120:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202392.449125:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202392.449136:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202392.449151:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442867840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202392.449155:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442867712:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202392.449173:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202392.449177:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202392.449180:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202392.449185:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1663 00000100:00080000:2.0:1788202392.449189:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202392.449194:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202392.449200:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202392.449213:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442867968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202392.449225:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788202392.449228:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788202392.449232:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202392.449236:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202392.449242:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1663 00000100:00080000:2.0:1788202392.449247:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202392.449252:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202392.449257:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202392.449271:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442868096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202392.449281:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202392/real 1788202392] req@ffff90234598dc00 x1875064442867712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202392.449304:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442867712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202392.449317:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202392.449321:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202392.449326:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442867712:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202392.449350:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442867840:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202392.449370:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202392.449380:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202392/real 1788202392] req@ffff90234598c700 x1875064442867840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202392.449388:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442867840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202392.449394:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202392.449396:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202392.449399:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442867840:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202392.449405:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442867968:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202392.449415:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202392.449422:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202392/real 1788202392] req@ffff90234598ce00 x1875064442867968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202392.449429:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442867968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202392.449435:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202392.449436:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202392.449438:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442867968:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202392.449442:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442868096:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202392.449452:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202392.449457:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202392/real 1788202392] req@ffff90234598e300 x1875064442868096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202392.449464:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442868096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202392.449469:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202392.449471:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202392.449472:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442868096:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202393.281051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202394.305110:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202395.329088:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202396.353077:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202397.377069:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202397.569005:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202397.569014:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202397.569018:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202397.569028:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1669 00000100:00080000:0.0:1788202397.569039:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202397.569046:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202397.569093:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442868224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202397.569114:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202397.569117:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202397.569119:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202397.569121:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1669 00000100:00080000:0.0:1788202397.569124:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202397.569127:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202397.569134:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202397.569144:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442868352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202397.569151:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202397.569154:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202397.569155:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202397.569158:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1669 00000100:00080000:0.0:1788202397.569161:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202397.569164:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202397.569167:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202397.569175:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442868224:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202397.569176:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442868480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202397.569191:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202397.569193:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202397.569201:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202397.569204:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1669 00000100:00080000:0.0:1788202397.569206:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202397.569209:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202397.569213:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202397.569220:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442868608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202397.569248:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202397.569291:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202397/real 1788202397] req@ffff9023450ba680 x1875064442868224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202397.569310:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442868224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202397.569320:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202397.569325:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202397.569329:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442868224:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202397.569337:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442868352:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202397.569361:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202397.569375:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202397/real 1788202397] req@ffff9023450b8e00 x1875064442868352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202397.569387:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442868352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202397.569396:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202397.569399:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202397.569403:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442868352:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202397.569426:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442868480:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202397.569446:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202397.569456:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202397/real 1788202397] req@ffff9023450bbb80 x1875064442868480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202397.569474:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442868480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202397.569483:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202397.569486:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202397.569490:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442868480:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202397.569496:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442868608:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202397.569514:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202397.569526:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202397/real 1788202397] req@ffff9023450bb480 x1875064442868608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202397.569537:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442868608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202397.569545:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202397.569548:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202397.569551:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442868608:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202398.401059:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202399.425018:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202400.449068:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202401.473076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202402.497037:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202402.689025:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202402.689036:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202402.689041:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202402.689051:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1674 00000100:00080000:2.0:1788202402.689063:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202402.689070:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202402.689109:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442868736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202402.689156:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202402.689160:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202402.689162:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202402.689167:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1674 00000100:00080000:2.0:1788202402.689171:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202402.689175:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202402.689185:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202402.689201:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442868864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202402.689213:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788202402.689218:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442868736:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202402.689218:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202402.689222:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202402.689232:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1674 00000100:00080000:2.0:1788202402.689236:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202402.689240:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202402.689246:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202402.689261:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442868992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202402.689271:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202402.689275:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202402.689278:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202402.689281:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1674 00000100:00080000:2.0:1788202402.689285:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202402.689292:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:1.0:1788202402.689293:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:2.0:1788202402.689300:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202402.689315:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442869120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202402.689352:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202402/real 1788202402] req@ffff90234598f480 x1875064442868736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202402.689380:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442868736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202402.689402:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202402.689407:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202402.689411:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442868736:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202402.689436:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442868864:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202402.689462:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202402.689477:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202402/real 1788202402] req@ffff90234598c380 x1875064442868864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202402.689490:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442868864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202402.689499:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202402.689502:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202402.689506:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442868864:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202402.689512:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442868992:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202402.689534:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202402.689546:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202402/real 1788202402] req@ffff90234598d180 x1875064442868992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202402.689558:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442868992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202402.689573:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202402.689576:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202402.689580:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442868992:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202402.689585:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442869120:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202402.689605:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202402.689617:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202402/real 1788202402] req@ffff90234598df80 x1875064442869120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202402.689630:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442869120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202402.689638:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202402.689641:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202402.689644:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442869120:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202403.521049:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202404.545061:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202405.569051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202406.593060:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202407.617026:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202407.809036:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202407.809043:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202407.809046:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202407.809056:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1679 00000100:00080000:0.0:1788202407.809065:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202407.809071:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202407.809102:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442869248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202407.809129:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202407.809132:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202407.809134:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202407.809136:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1679 00000100:00080000:0.0:1788202407.809139:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202407.809142:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202407.809150:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202407.809161:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442869248:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202407.809162:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442869376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202407.809175:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202407.809178:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202407.809180:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202407.809184:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1679 00000100:00080000:0.0:1788202407.809188:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202407.809192:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202407.809198:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202407.809214:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442869504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202407.809225:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788202407.809226:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202407.809232:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202407.809234:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202407.809239:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1679 00000100:00080000:0.0:1788202407.809245:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202407.809248:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202407.809253:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202407.809263:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202407/real 1788202407] req@ffff9023450bb480 x1875064442869248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202407.809267:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442869632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202407.809289:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442869248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202407.809298:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202407.809301:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202407.809305:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442869248:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202407.809328:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442869376:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202407.809353:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202407.809367:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202407/real 1788202407] req@ffff9023450bbb80 x1875064442869376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202407.809380:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442869376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202407.809389:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202407.809393:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202407.809397:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442869376:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202407.809403:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442869504:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202407.809422:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202407.809431:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202407/real 1788202407] req@ffff9023450b8e00 x1875064442869504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202407.809440:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442869504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202407.809447:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202407.809449:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202407.809451:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442869504:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202407.809454:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442869632:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202407.809466:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202407.809474:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202407/real 1788202407] req@ffff9023450ba680 x1875064442869632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202407.809482:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442869632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202407.809488:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202407.809490:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202407.809491:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442869632:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202408.641048:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202409.665039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202410.689070:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202411.713012:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202412.737106:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202412.928997:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202412.929006:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202412.929011:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202412.929020:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1684 00000100:00080000:0.0:1788202412.929031:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202412.929038:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202412.929074:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442869760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202412.929105:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202412.929109:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202412.929111:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202412.929114:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1684 00000100:00080000:0.0:1788202412.929118:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202412.929122:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202412.929131:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202412.929143:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442869888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202412.929159:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202412.929164:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202412.929166:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202412.929170:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1684 00000100:00080000:0.0:1788202412.929173:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202412.929181:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202412.929189:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202412.929189:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442869760:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202412.929210:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442870016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202412.929223:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202412.929227:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202412.929229:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202412.929233:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1684 00000100:00080000:0.0:1788202412.929237:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202412.929240:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202412.929245:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202412.929259:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442870144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202412.929293:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202412.929351:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202412/real 1788202412] req@ffff9023450b8380 x1875064442869760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202412.929374:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442869760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202412.929387:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202412.929392:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202412.929397:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442869760:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202412.929408:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442869888:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202412.929431:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202412.929442:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202412/real 1788202412] req@ffff9023450b8a80 x1875064442869888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202412.929453:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442869888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202412.929468:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202412.929472:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202412.929476:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442869888:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202412.929524:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442870016:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202412.929548:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202412.929557:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202412/real 1788202412] req@ffff9023450bb800 x1875064442870016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202412.929566:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442870016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202412.929572:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202412.929574:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202412.929576:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442870016:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202412.929580:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442870144:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202412.929592:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202412.929598:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202412/real 1788202412] req@ffff9023450b9500 x1875064442870144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202412.929605:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442870144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202412.929610:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202412.929612:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202412.929615:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442870144:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202413.761111:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202414.785115:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202415.809116:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202416.833062:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202417.857038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202418.049044:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202418.049059:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202418.049065:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202418.049084:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1689 00000100:00080000:2.0:1788202418.049099:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202418.049109:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202418.049165:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442870272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202418.049203:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202418.049206:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202418.049208:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202418.049212:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1689 00000100:00080000:2.0:1788202418.049215:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202418.049218:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202418.049229:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202418.049241:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442870400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202418.049251:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202418.049254:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202418.049256:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202418.049259:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1689 00000100:00080000:2.0:1788202418.049261:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202418.049265:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202418.049269:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202418.049278:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442870528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202418.049285:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442870272:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202418.049292:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202418.049298:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202418.049299:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202418.049302:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1689 00000100:00080000:2.0:1788202418.049304:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202418.049307:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202418.049312:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202418.049324:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442870656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202418.049368:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202418.049438:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202418/real 1788202418] req@ffff90234598fb80 x1875064442870272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202418.049455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442870272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202418.049463:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202418.049467:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202418.049470:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442870272:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202418.049476:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442870400:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202418.049495:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202418.049504:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202418/real 1788202418] req@ffff90234598ed80 x1875064442870400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202418.049512:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442870400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202418.049518:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202418.049520:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202418.049522:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442870400:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202418.049551:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442870528:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202418.049565:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202418.049572:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202418/real 1788202418] req@ffff90234598f100 x1875064442870528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202418.049585:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442870528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202418.049591:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202418.049593:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202418.049595:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442870528:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202418.049600:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442870656:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202418.049610:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202418.049616:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202418/real 1788202418] req@ffff90234598df80 x1875064442870656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202418.049630:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442870656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202418.049636:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202418.049638:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202418.049640:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442870656:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202418.881062:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202419.905118:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202420.929081:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202421.953077:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202422.977124:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202423.169006:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202423.169017:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202423.169021:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202423.169032:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1694 00000100:00080000:2.0:1788202423.169044:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202423.169052:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202423.169114:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442870784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202423.169143:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202423.169146:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202423.169148:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202423.169151:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1694 00000100:00080000:2.0:1788202423.169154:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202423.169157:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202423.169168:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202423.169182:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442870912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202423.169197:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202423.169201:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202423.169203:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788202423.169208:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442870784:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202423.169209:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1694 00000100:00080000:2.0:1788202423.169218:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202423.169231:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202423.169235:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202423.169248:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442871040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202423.169255:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202423.169258:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202423.169259:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202423.169263:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1694 00000100:00080000:2.0:1788202423.169266:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202423.169269:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202423.169272:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202423.169281:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442871168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202423.169295:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202423.169371:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202423/real 1788202423] req@ffff90234598d180 x1875064442870784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202423.169399:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442870784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202423.169410:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202423.169415:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202423.169420:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442870784:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202423.169446:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442870912:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202423.169470:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202423.169482:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202423/real 1788202423] req@ffff90234598c380 x1875064442870912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202423.169496:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442870912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202423.169505:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202423.169508:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202423.169512:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442870912:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202423.169518:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442871040:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202423.169545:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202423.169559:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202423/real 1788202423] req@ffff90234598f480 x1875064442871040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202423.169573:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442871040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202423.169588:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202423.169591:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202423.169594:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442871040:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202423.169601:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442871168:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202423.169620:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202423.169635:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202423/real 1788202423] req@ffff90234598e300 x1875064442871168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202423.169647:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442871168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202423.169655:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202423.169658:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202423.169661:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442871168:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202424.001050:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202425.025085:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202426.049041:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202427.073023:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202428.097030:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202428.289021:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202428.289032:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202428.289036:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202428.289046:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1699 00000100:00080000:3.0:1788202428.289057:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202428.289064:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202428.289101:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442871296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202428.289137:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202428.289141:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202428.289142:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202428.289146:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1699 00000100:00080000:3.0:1788202428.289150:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202428.289153:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202428.289162:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202428.289176:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442871424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202428.289184:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442871296:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202428.289190:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202428.289194:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202428.289196:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202428.289201:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1699 00000100:00080000:3.0:1788202428.289205:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202428.289208:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202428.289213:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202428.289238:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442871552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202428.289246:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202428.289251:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202428.289252:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202428.289255:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1699 00000100:00080000:3.0:1788202428.289266:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202428.289270:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202428.289275:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202428.289284:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202428.289289:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442871680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202428.289329:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202428/real 1788202428] req@ffff902352688380 x1875064442871296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202428.289348:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442871296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202428.289363:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202428.289367:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202428.289371:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442871296:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202428.289394:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442871424:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202428.289415:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202428.289426:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202428/real 1788202428] req@ffff902352689880 x1875064442871424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202428.289438:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442871424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202428.289446:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202428.289449:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202428.289452:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442871424:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202428.289458:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442871552:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202428.289475:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202428.289485:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202428/real 1788202428] req@ffff90235268bb80 x1875064442871552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202428.289496:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442871552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202428.289504:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202428.289507:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202428.289511:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442871552:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202428.289516:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442871680:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202428.289546:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202428.289556:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202428/real 1788202428] req@ffff902352688700 x1875064442871680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202428.289567:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442871680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202428.289574:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202428.289577:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202428.289580:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442871680:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202429.121030:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202430.145050:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202431.168992:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202432.193129:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202433.217076:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202433.408968:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202433.408977:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202433.408980:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202433.408988:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1704 00000100:00080000:3.0:1788202433.408996:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202433.409001:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202433.409031:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442871808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202433.409051:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202433.409053:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202433.409054:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202433.409057:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1704 00000100:00080000:3.0:1788202433.409059:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202433.409061:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202433.409068:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202433.409076:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442871936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202433.409083:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202433.409084:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202433.409085:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202433.409087:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1704 00000100:00080000:3.0:1788202433.409089:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202433.409091:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202433.409093:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202433.409101:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442871808:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202433.409101:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442872064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202433.409110:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202433.409115:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202433.409116:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202433.409118:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1704 00000100:00080000:3.0:1788202433.409120:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202433.409123:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202433.409127:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202433.409161:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442872192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202433.409181:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202433.409224:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202433/real 1788202433] req@ffff902352689180 x1875064442871808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202433.409239:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442871808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202433.409246:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202433.409249:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202433.409252:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442871808:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202433.409257:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442871936:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202433.409272:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202433.409280:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202433/real 1788202433] req@ffff902352689500 x1875064442871936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202433.409294:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442871936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202433.409299:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202433.409301:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202433.409303:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442871936:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202433.409319:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442872064:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202433.409330:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202433.409337:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202433/real 1788202433] req@ffff90235268a680 x1875064442872064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202433.409343:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442872064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202433.409348:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202433.409350:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202433.409352:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442872064:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202433.409355:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442872192:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202433.409363:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202433.409368:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202433/real 1788202433] req@ffff902352689c00 x1875064442872192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202433.409378:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442872192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202433.409382:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202433.409383:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202433.409385:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442872192:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202434.241090:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202435.265065:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202436.289108:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202437.313122:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202438.337042:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202438.528957:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202438.528966:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202438.528969:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202438.528978:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1710 00000100:00080000:2.0:1788202438.528988:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202438.528994:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202438.529033:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442872320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202438.529066:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202438.529069:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202438.529071:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202438.529074:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1710 00000100:00080000:2.0:1788202438.529078:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202438.529081:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202438.529088:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202438.529099:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442872448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202438.529109:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202438.529114:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788202438.529116:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442872320:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202438.529116:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202438.529126:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1710 00000100:00080000:2.0:1788202438.529129:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202438.529132:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202438.529136:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202438.529146:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442872576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202438.529153:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202438.529156:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202438.529157:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202438.529160:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1710 00000100:00080000:2.0:1788202438.529170:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202438.529173:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202438.529177:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202438.529185:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442872704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202438.529223:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442872448:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202438.529245:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442872576:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202438.529261:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442872704:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202439.361027:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202440.384995:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202441.409011:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788202441.601175:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:0.0:1788202441.601221:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788202441.601234:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788202441.601267:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788202441.601273:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788202441.601278:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:1.0:1788202441.601323:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202438/real 1788202441] req@ffff90234598c700 x1875064442872448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202441.601350:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442872448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202441.601362:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202441.601368:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202441.601374:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442872448:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202441.601383:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202438/real 1788202441] req@ffff90234598dc00 x1875064442872576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202441.601395:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442872576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202441.601403:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202441.601407:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202441.601410:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442872576:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202441.601417:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202438/real 1788202441] req@ffff90234598f800 x1875064442872704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202441.601428:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442872704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202441.601436:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202441.601439:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202441.601442:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442872704:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202441.601448:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202438/real 1788202441] req@ffff90234598ce00 x1875064442872320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202441.601459:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442872320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202441.601466:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202441.601469:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202441.601473:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442872320:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788202442.433048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202443.457072:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202443.649028:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202443.649036:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202443.649040:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202443.649049:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1715 00000100:00080000:2.0:1788202443.649060:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202443.649067:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202443.649109:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442872832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202443.649139:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202443.649144:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202443.649147:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202443.649152:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1715 00000100:00080000:2.0:1788202443.649156:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202443.649162:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202443.649173:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202443.649194:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442872832:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202443.649195:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064442872960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202443.649241:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202443.649245:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202443.649248:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202443.649254:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1715 00000100:00080000:2.0:1788202443.649259:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202443.649265:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202443.649281:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202443.649301:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064442873088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202443.649315:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788202443.649328:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202443.649334:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202443.649337:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202443.649347:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1715 00000100:00080000:2.0:1788202443.649353:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202443.649361:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202443.649376:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202443.649385:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202443/real 1788202443] req@ffff90234598d880 x1875064442872832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202443.649402:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442873216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202443.649415:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442872832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202443.649439:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202443.649446:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202443.649453:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442872832:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202443.649489:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442872960:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202443.649521:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202443.649537:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202443/real 1788202443] req@ffff90234598e680 x1875064442872960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202443.649551:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064442872960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202443.649559:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202443.649562:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202443.649566:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442872960:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202443.649574:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442873088:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202443.649600:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202443.649616:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202443/real 1788202443] req@ffff90234598ca80 x1875064442873088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202443.649632:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064442873088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202443.649643:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202443.649646:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202443.649649:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442873088:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202443.649656:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442873216:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202443.649676:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202443.649688:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202443/real 1788202443] req@ffff90234598ce00 x1875064442873216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202443.649699:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442873216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202443.649706:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202443.649709:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202443.649712:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442873216:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202444.481094:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202445.505048:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202446.529043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202447.553037:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202448.576991:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202448.769020:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202448.769031:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202448.769036:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202448.769047:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1720 00000100:00080000:3.0:1788202448.769059:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202448.769066:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202448.769104:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442873344/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202448.769134:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202448.769139:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202448.769141:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202448.769145:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1720 00000100:00080000:3.0:1788202448.769150:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202448.769154:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202448.769164:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202448.769176:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442873472/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202448.769185:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202448.769187:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202448.769189:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202448.769193:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1720 00000100:00080000:3.0:1788202448.769197:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788202448.769203:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442873344:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202448.769206:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202448.769216:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202448.769230:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442873600/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202448.769242:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202448.769245:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202448.769247:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202448.769250:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1720 00000100:00080000:3.0:1788202448.769252:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202448.769256:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202448.769259:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202448.769268:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442873728/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202448.769284:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202448.769336:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202448/real 1788202448] req@ffff902352688e00 x1875064442873344/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202448.769358:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442873344/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202448.769369:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202448.769375:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202448.769379:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442873344:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202448.769403:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442873472:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202448.769428:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202448.769441:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202448/real 1788202448] req@ffff90235268a300 x1875064442873472/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202448.769455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442873472/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202448.769464:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202448.769468:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202448.769471:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442873472:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202448.769477:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442873600:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202448.769506:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202448.769519:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202448/real 1788202448] req@ffff902352688000 x1875064442873600/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202448.769531:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442873600/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202448.769546:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202448.769549:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202448.769553:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442873600:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202448.769559:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442873728:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202448.769577:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202448.769587:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202448/real 1788202448] req@ffff90235268b800 x1875064442873728/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202448.769599:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442873728/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202448.769608:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202448.769611:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202448.769614:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442873728:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202449.601043:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202450.625001:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202451.649043:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202452.673026:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202453.697037:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202453.888968:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202453.888979:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202453.888984:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202453.888995:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1725 00000100:00080000:0.0:1788202453.889006:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202453.889013:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202453.889053:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442873856/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202453.889085:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202453.889089:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202453.889090:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202453.889094:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1725 00000100:00080000:0.0:1788202453.889098:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202453.889103:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202453.889112:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202453.889127:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442873984/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202453.889139:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202453.889143:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202453.889145:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202453.889150:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1725 00000100:00080000:0.0:1788202453.889154:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202453.889161:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202453.889164:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442873856:192.168.201.130@tcp:250:kworker.0 00000008:00100000:0.0:1788202453.889168:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202453.889185:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442874112/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202453.889196:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202453.889200:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202453.889201:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202453.889204:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1725 00000100:00080000:0.0:1788202453.889220:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202453.889224:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202453.889229:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202453.889235:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202453.889239:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442874240/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202453.889281:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202453/real 1788202453] req@ffff9023450b8700 x1875064442873856/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202453.889298:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442873856/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202453.889306:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202453.889309:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202453.889312:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442873856:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202453.889317:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442873984:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202453.889333:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202453.889341:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202453/real 1788202453] req@ffff9023450b9c00 x1875064442873984/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202453.889349:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442873984/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202453.889355:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202453.889357:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202453.889359:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442873984:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202453.889377:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442874112:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202453.889388:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202453.889394:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202453/real 1788202453] req@ffff9023450b9f80 x1875064442874112/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202453.889401:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442874112/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202453.889407:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202453.889409:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202453.889411:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442874112:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202453.889415:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442874240:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202453.889424:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202453.889429:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202453/real 1788202453] req@ffff9023450bad80 x1875064442874240/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202453.889436:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442874240/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202453.889441:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202453.889443:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202453.889446:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442874240:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202454.721060:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202455.745035:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202456.769064:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202457.793090:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202458.817058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202459.008991:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202459.009001:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202459.009005:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202459.009015:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1730 00000100:00080000:0.0:1788202459.009026:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202459.009033:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202459.009069:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442874368/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202459.009097:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202459.009101:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202459.009102:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202459.009106:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1730 00000100:00080000:0.0:1788202459.009110:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202459.009114:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202459.009123:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202459.009135:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442874496/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202459.009144:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442874368:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202459.009149:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202459.009155:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202459.009157:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202459.009161:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1730 00000100:00080000:0.0:1788202459.009165:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202459.009169:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202459.009174:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202459.009186:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442874624/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202459.009195:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202459.009199:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202459.009201:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202459.009205:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1730 00000100:00080000:0.0:1788202459.009209:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788202459.009213:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788202459.009217:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202459.009223:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202459.009236:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442874752/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202459.009253:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202459/real 1788202459] req@ffff9023450b8000 x1875064442874368/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202459.009269:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442874368/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202459.009276:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202459.009279:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202459.009282:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442874368:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202459.009300:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442874496:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202459.009315:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202459.009323:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202459/real 1788202459] req@ffff9023450ba300 x1875064442874496/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202459.009330:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442874496/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202459.009335:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202459.009337:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202459.009348:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442874496:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202459.009354:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442874624:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202459.009366:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202459.009373:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202459/real 1788202459] req@ffff9023450baa00 x1875064442874624/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202459.009381:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442874624/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202459.009386:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202459.009388:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202459.009390:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442874624:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202459.009393:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442874752:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202459.009407:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202459.009414:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202459/real 1788202459] req@ffff9023450b9180 x1875064442874752/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202459.009421:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442874752/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202459.009426:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202459.009428:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202459.009429:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442874752:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202459.841051:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202460.865085:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202461.889057:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202462.912998:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202463.937023:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202464.128949:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202464.128961:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202464.128965:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202464.128975:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1735 00000100:00080000:0.0:1788202464.128992:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202464.129001:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202464.129041:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442874880/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202464.129072:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202464.129078:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202464.129080:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202464.129084:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1735 00000100:00080000:0.0:1788202464.129094:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202464.129100:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202464.129110:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202464.129121:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442874880:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202464.129128:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442875008/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202464.129143:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202464.129146:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202464.129148:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202464.129152:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1735 00000100:00080000:0.0:1788202464.129157:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202464.129161:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202464.129166:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202464.129179:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442875136/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202464.129183:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788202464.129194:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202464.129198:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202464.129200:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202464.129205:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1735 00000100:00080000:0.0:1788202464.129209:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202464.129213:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202464.129219:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202464.129227:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202464/real 1788202464] req@ffff9023450b9180 x1875064442874880/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202464.129233:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442875264/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202464.129246:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442874880/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202464.129256:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202464.129259:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202464.129262:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442874880:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202464.129299:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442875008:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202464.129324:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202464.129337:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202464/real 1788202464] req@ffff9023450baa00 x1875064442875008/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202464.129350:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442875008/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202464.129356:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202464.129358:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202464.129361:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442875008:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202464.129365:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442875136:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202464.129380:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202464.129387:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202464/real 1788202464] req@ffff9023450ba300 x1875064442875136/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202464.129399:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442875136/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202464.129404:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202464.129406:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202464.129408:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442875136:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202464.129412:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442875264:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202464.129421:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202464.129427:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202464/real 1788202464] req@ffff9023450b8000 x1875064442875264/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202464.129434:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442875264/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202464.129439:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202464.129441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202464.129443:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442875264:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202464.961059:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202465.985000:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202467.009000:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202468.033030:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202469.057035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202469.248980:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202469.248989:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202469.248992:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202469.249000:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1740 00000100:00080000:0.0:1788202469.249009:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202469.249015:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202469.249050:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442875392/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202469.249074:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202469.249077:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202469.249079:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202469.249081:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1740 00000100:00080000:0.0:1788202469.249084:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202469.249086:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202469.249093:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202469.249102:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442875520/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202469.249109:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202469.249112:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202469.249113:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202469.249116:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1740 00000100:00080000:0.0:1788202469.249119:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202469.249122:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202469.249125:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202469.249135:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442875648/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202469.249140:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442875392:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202469.249145:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202469.249151:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202469.249153:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202469.249157:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1740 00000100:00080000:0.0:1788202469.249161:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202469.249165:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202469.249170:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202469.249185:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442875776/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202469.249210:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202469.249250:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202469/real 1788202469] req@ffff9023450bad80 x1875064442875392/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202469.249267:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442875392/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202469.249278:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202469.249283:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202469.249287:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442875392:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202469.249295:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442875520:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202469.249315:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202469.249327:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202469/real 1788202469] req@ffff9023450b9f80 x1875064442875520/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202469.249340:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442875520/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202469.249348:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202469.249351:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202469.249354:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442875520:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202469.249377:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442875648:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202469.249393:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202469.249402:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202469/real 1788202469] req@ffff9023450b9c00 x1875064442875648/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202469.249414:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442875648/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202469.249422:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202469.249425:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202469.249428:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442875648:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202469.249433:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442875776:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202469.249448:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202469.249456:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202469/real 1788202469] req@ffff9023450b8700 x1875064442875776/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202469.249467:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442875776/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202469.249475:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202469.249478:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202469.249481:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442875776:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202470.081014:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202471.105045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202472.129108:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202473.153032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202474.177051:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202474.369023:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202474.369034:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202474.369038:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202474.369046:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1745 00000100:00080000:2.0:1788202474.369055:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202474.369060:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202474.369093:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442875904/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202474.369116:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202474.369119:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202474.369121:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202474.369124:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1745 00000100:00080000:2.0:1788202474.369126:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202474.369129:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202474.369138:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202474.369148:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442876032/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202474.369156:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202474.369158:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202474.369160:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788202474.369165:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442875904:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202474.369166:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1745 00000100:00080000:2.0:1788202474.369172:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202474.369176:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202474.369179:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202474.369189:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442876160/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202474.369195:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202474.369198:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202474.369199:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202474.369202:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1745 00000100:00080000:2.0:1788202474.369206:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202474.369209:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202474.369212:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202474.369219:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442876288/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202474.369228:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202474.369277:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202474/real 1788202474] req@ffff90234598f800 x1875064442875904/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202474.369297:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442875904/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202474.369304:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202474.369308:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202474.369310:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442875904:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202474.369329:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442876032:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202474.369345:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202474.369353:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202474/real 1788202474] req@ffff90234598dc00 x1875064442876032/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202474.369361:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442876032/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202474.369367:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202474.369369:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202474.369371:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442876032:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202474.369374:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442876160:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202474.369385:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202474.369390:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202474/real 1788202474] req@ffff90234598c700 x1875064442876160/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202474.369398:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442876160/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202474.369403:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202474.369405:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202474.369406:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442876160:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202474.369409:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442876288:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202474.369422:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202474.369464:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202474/real 1788202474] req@ffff90234598e300 x1875064442876288/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202474.369485:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442876288/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202474.369492:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202474.369494:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202474.369496:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442876288:192.168.201.130@tcp:8:kworker.0 00010000:00010000:3.0:1788202475.201036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202476.225256:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202477.249081:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202478.273127:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202479.296967:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202479.488946:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202479.488954:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202479.488957:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202479.488965:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1751 00000100:00080000:2.0:1788202479.488973:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202479.488978:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202479.489006:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442876416/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202479.489028:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202479.489031:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202479.489032:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202479.489035:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1751 00000100:00080000:2.0:1788202479.489037:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202479.489039:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202479.489045:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202479.489053:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442876544/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202479.489064:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202479.489068:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202479.489069:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202479.489072:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1751 00000100:00080000:2.0:1788202479.489074:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202479.489077:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442876416:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202479.489078:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202479.489085:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202479.489094:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442876672/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202479.489099:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202479.489101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202479.489102:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202479.489104:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1751 00000100:00080000:2.0:1788202479.489107:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202479.489109:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202479.489111:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202479.489118:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442876800/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202479.489142:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202479.489174:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202479/real 1788202479] req@ffff90234598f480 x1875064442876416/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202479.489187:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442876416/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202479.489194:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202479.489198:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202479.489201:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442876416:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202479.489221:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442876544:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202479.489238:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202479.489247:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202479/real 1788202479] req@ffff90234598c380 x1875064442876544/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202479.489255:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442876544/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202479.489261:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202479.489263:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202479.489265:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442876544:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202479.489269:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442876672:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202479.489284:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202479.489295:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202479/real 1788202479] req@ffff90234598d180 x1875064442876672/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202479.489305:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442876672/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202479.489310:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202479.489313:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202479.489315:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442876672:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202479.489319:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442876800:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202479.489331:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202479.489356:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202479/real 1788202479] req@ffff90234598df80 x1875064442876800/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202479.489373:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442876800/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202479.489381:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202479.489385:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202479.489391:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442876800:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202480.321046:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202481.345029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202482.369027:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202483.392947:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202484.417041:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202484.608968:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202484.608977:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202484.608980:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202484.608987:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1756 00000100:00080000:0.0:1788202484.608994:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202484.609000:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202484.609029:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442876928/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202484.609050:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202484.609054:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202484.609055:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202484.609058:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1756 00000100:00080000:0.0:1788202484.609060:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202484.609062:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202484.609069:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202484.609077:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442877056/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202484.609084:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202484.609086:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202484.609087:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202484.609090:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1756 00000100:00080000:0.0:1788202484.609091:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202484.609093:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202484.609096:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202484.609102:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442876928:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202484.609104:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442877184/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202484.609115:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202484.609116:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202484.609117:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202484.609119:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1756 00000100:00080000:0.0:1788202484.609121:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202484.609130:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202484.609136:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202484.609144:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442877312/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202484.609169:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202484.609211:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202484/real 1788202484] req@ffff9023450b9500 x1875064442876928/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202484.609227:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442876928/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202484.609237:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202484.609241:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202484.609257:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442876928:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202484.609265:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442877056:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202484.609287:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202484.609299:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202484/real 1788202484] req@ffff9023450bb800 x1875064442877056/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202484.609310:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442877056/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202484.609317:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202484.609320:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202484.609323:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442877056:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202484.609346:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442877184:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202484.609363:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202484.609373:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202484/real 1788202484] req@ffff9023450b8a80 x1875064442877184/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202484.609384:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442877184/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202484.609391:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202484.609394:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202484.609397:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442877184:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202484.609402:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442877312:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202484.609416:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202484.609424:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202484/real 1788202484] req@ffff9023450b8380 x1875064442877312/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202484.609434:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442877312/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202484.609441:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202484.609443:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202484.609446:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442877312:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202485.441001:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202486.465036:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202487.488971:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202488.513026:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202489.537017:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202489.729033:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202489.729041:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202489.729044:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202489.729053:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1761 00000100:00080000:0.0:1788202489.729063:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202489.729068:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202489.729102:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442877440/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202489.729129:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202489.729133:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202489.729135:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202489.729146:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1761 00000100:00080000:0.0:1788202489.729150:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202489.729154:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202489.729218:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442877440:192.168.201.130@tcp:250:kworker.0 00000800:00000100:1.0:1788202489.729272:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:0.0:1788202489.729294:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202489.729307:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202489/real 1788202489] req@ffff9023450ba680 x1875064442877440/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202489.729312:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442877568/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202489.729334:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202489.729336:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202489.729340:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788202489.729340:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442877440/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202489.729348:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1761 00000100:00080000:0.0:1788202489.729355:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202489.729359:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202489.729360:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202489.729368:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1788202489.729369:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202489.729374:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442877440:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202489.729381:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442877696/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202489.729392:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202489.729396:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202489.729398:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788202489.729402:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442877568:192.168.201.130@tcp:38:kworker.0 00000100:00080000:0.0:1788202489.729407:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1761 00000100:00080000:0.0:1788202489.729414:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202489.729418:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202489.729423:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202489.729438:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442877824/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202489.729449:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202489.729463:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202489/real 1788202489] req@ffff9023450b8e00 x1875064442877568/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202489.729474:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442877568/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202489.729481:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202489.729483:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202489.729486:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442877568:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202489.729490:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442877696:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202489.729504:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202489.729510:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202489/real 1788202489] req@ffff9023450bbb80 x1875064442877696/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202489.729518:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442877696/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202489.729528:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202489.729530:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202489.729532:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442877696:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202489.729541:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442877824:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202489.729551:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202489.729557:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202489/real 1788202489] req@ffff9023450bb480 x1875064442877824/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202489.729564:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442877824/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202489.729569:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202489.729571:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202489.729573:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442877824:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202490.561046:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202491.585018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202492.609013:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202493.633040:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202494.657086:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202494.848994:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202494.849002:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202494.849005:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202494.849015:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1766 00000100:00080000:2.0:1788202494.849024:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202494.849030:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202494.849090:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442877952/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202494.849120:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202494.849123:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202494.849125:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202494.849128:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1766 00000100:00080000:2.0:1788202494.849130:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202494.849133:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202494.849140:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202494.849154:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442878080/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202494.849163:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202494.849165:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202494.849166:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202494.849171:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1766 00000100:00080000:2.0:1788202494.849178:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202494.849180:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442877952:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202494.849183:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202494.849199:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202494.849207:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442878208/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202494.849216:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202494.849218:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202494.849220:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202494.849223:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1766 00000100:00080000:2.0:1788202494.849226:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202494.849230:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202494.849233:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202494.849243:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442878336/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202494.849270:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202494.849320:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202494/real 1788202494] req@ffff90234598f100 x1875064442877952/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202494.849337:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442877952/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202494.849347:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202494.849351:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202494.849355:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442877952:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202494.849393:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442878080:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202494.849418:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202494.849431:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202494/real 1788202494] req@ffff90234598ed80 x1875064442878080/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202494.849444:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442878080/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202494.849452:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202494.849455:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202494.849457:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442878080:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202494.849462:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442878208:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202494.849486:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202494.849493:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202494/real 1788202494] req@ffff90234598fb80 x1875064442878208/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202494.849501:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442878208/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202494.849506:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202494.849509:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202494.849511:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442878208:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202494.849515:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442878336:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202494.849526:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202494.849532:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202494/real 1788202494] req@ffff90234598df80 x1875064442878336/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202494.849539:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442878336/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202494.849543:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202494.849545:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202494.849547:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442878336:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202495.681089:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202496.705072:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202497.729119:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202498.753061:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202499.777068:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202499.969017:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202499.969028:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202499.969032:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202499.969040:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1771 00000100:00080000:0.0:1788202499.969049:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202499.969055:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202499.969091:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442878464/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202499.969112:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202499.969115:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202499.969116:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202499.969119:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1771 00000100:00080000:0.0:1788202499.969121:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202499.969124:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202499.969133:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202499.969147:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442878592/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202499.969158:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202499.969162:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202499.969164:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202499.969169:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1771 00000100:00080000:0.0:1788202499.969172:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202499.969179:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202499.969180:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442878464:192.168.201.130@tcp:250:kworker.0 00000008:00100000:0.0:1788202499.969187:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202499.969203:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442878720/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202499.969216:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202499.969220:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202499.969222:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202499.969227:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1771 00000100:00080000:0.0:1788202499.969239:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202499.969245:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202499.969251:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202499.969252:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202499.969283:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442878848/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202499.969312:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202499/real 1788202499] req@ffff9023450bb480 x1875064442878464/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202499.969341:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442878464/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202499.969354:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202499.969359:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202499.969363:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442878464:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202499.969369:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442878592:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202499.969388:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202499.969398:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202499/real 1788202499] req@ffff9023450bbb80 x1875064442878592/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202499.969411:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442878592/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202499.969417:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202499.969419:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202499.969421:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442878592:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202499.969439:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442878720:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202499.969450:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202499.969457:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202499/real 1788202499] req@ffff9023450b8e00 x1875064442878720/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202499.969465:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442878720/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202499.969470:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202499.969471:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202499.969473:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442878720:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202499.969477:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442878848:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202499.969486:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202499.969492:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202499/real 1788202499] req@ffff9023450ba680 x1875064442878848/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202499.969499:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442878848/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202499.969508:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202499.969509:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202499.969511:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442878848:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202500.801047:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202501.825062:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202502.849029:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202503.873097:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202504.897106:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202505.089035:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202505.089047:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202505.089052:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202505.089062:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1776 00000100:00080000:2.0:1788202505.089074:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202505.089082:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202505.089131:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442878976/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202505.089161:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202505.089164:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202505.089166:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202505.089170:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1776 00000100:00080000:2.0:1788202505.089173:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202505.089176:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202505.089184:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202505.089199:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442879104/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202505.089213:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442878976:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202505.089222:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202505.089227:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202505.089229:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202505.089235:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1776 00000100:00080000:2.0:1788202505.089239:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202505.089243:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202505.089249:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202505.089264:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442879232/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202505.089273:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202505.089277:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202505.089279:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202505.089283:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1776 00000100:00080000:2.0:1788202505.089287:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202505.089292:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202505.089298:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202505.089310:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442879360/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202505.089326:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442879104:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202505.089372:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442879232:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202505.089400:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442879360:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202505.921080:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202506.945032:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202507.969005:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788202508.161308:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788202508.161355:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788202508.161374:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202508.161423:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202508.161434:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202508.161444:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:1.0:1788202508.161729:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202505/real 1788202508] req@ffff90234598c380 x1875064442879104/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202508.161775:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442879104/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202508.161789:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202508.161795:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202508.161799:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442879104:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202508.161808:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202505/real 1788202508] req@ffff90234598f480 x1875064442879232/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202508.161831:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442879232/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202508.161854:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202508.161858:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202508.161862:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442879232:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202508.161868:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202505/real 1788202508] req@ffff90234598e300 x1875064442879360/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202508.161918:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442879360/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202508.161933:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202508.161936:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202508.161939:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442879360:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202508.161946:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202505/real 1788202508] req@ffff90234598d180 x1875064442878976/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202508.161960:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442878976/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202508.161968:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202508.161971:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202508.161975:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442878976:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788202508.993016:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202510.017001:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202510.208953:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202510.208962:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202510.208965:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202510.208974:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1781 00000100:00080000:2.0:1788202510.208990:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202510.208996:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202510.209028:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442879488/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202510.209051:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202510.209054:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202510.209055:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202510.209059:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1781 00000100:00080000:2.0:1788202510.209062:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202510.209066:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202510.209075:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202510.209089:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442879616/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202510.209100:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202510.209104:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202510.209106:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202510.209111:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1781 00000100:00080000:2.0:1788202510.209116:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202510.209119:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442879488:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202510.209124:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202510.209133:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202510.209148:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442879744/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202510.209160:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202510.209164:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202510.209166:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202510.209170:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1781 00000100:00080000:2.0:1788202510.209182:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788202510.209187:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788202510.209190:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202510.209198:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202510.209212:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442879872/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202510.209233:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202510/real 1788202510] req@ffff90234598c700 x1875064442879488/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202510.209256:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442879488/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202510.209281:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202510.209286:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202510.209290:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442879488:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202510.209298:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442879616:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202510.209324:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202510.209338:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202510/real 1788202510] req@ffff90234598dc00 x1875064442879616/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202510.209350:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442879616/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202510.209358:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202510.209361:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202510.209364:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442879616:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202510.209403:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442879744:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202510.209430:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202510.209444:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202510/real 1788202510] req@ffff90234598f800 x1875064442879744/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202510.209464:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442879744/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202510.209473:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202510.209477:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202510.209482:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442879744:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202510.209488:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442879872:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202510.209508:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202510.209520:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202510/real 1788202510] req@ffff90234598ce00 x1875064442879872/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202510.209531:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442879872/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202510.209539:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202510.209542:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202510.209546:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442879872:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202511.041052:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202512.065092:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202513.089255:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202514.113071:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202515.137064:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202515.329041:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202515.329052:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202515.329057:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202515.329069:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1786 00000100:00080000:2.0:1788202515.329083:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202515.329092:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202515.329137:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064442880000/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202515.329167:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202515.329172:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202515.329174:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202515.329179:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1786 00000100:00080000:2.0:1788202515.329184:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202515.329190:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202515.329204:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202515.329226:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064442880128/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202515.329233:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442880000:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202515.329260:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202515.329264:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202515.329267:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202515.329272:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1786 00000100:00080000:2.0:1788202515.329276:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202515.329282:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202515.329291:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202515.329310:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442880256/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202515.329328:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202515.329333:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202515.329335:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202515.329341:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1786 00000100:00080000:2.0:1788202515.329345:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202515.329348:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202515.329361:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202515.329371:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202515.329375:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442880384/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202515.329410:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202515/real 1788202515] req@ffff90234598ca80 x1875064442880000/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202515.329537:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064442880000/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202515.329548:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202515.329553:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202515.329558:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442880000:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202515.329618:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442880128:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202515.329649:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202515.329660:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202515/real 1788202515] req@ffff90234598e680 x1875064442880128/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202515.329669:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064442880128/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202515.329676:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202515.329678:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202515.329680:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442880128:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202515.329685:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442880256:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202515.329698:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202515.329704:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202515/real 1788202515] req@ffff90234598d880 x1875064442880256/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202515.329732:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442880256/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202515.329739:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202515.329742:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202515.329744:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442880256:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202515.329749:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442880384:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202515.329765:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202515.329773:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202515/real 1788202515] req@ffff90234598ce00 x1875064442880384/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202515.329781:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442880384/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202515.329786:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202515.329789:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202515.329791:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442880384:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202516.161082:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202517.185095:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202518.209107:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202519.233060:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202520.257013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202520.449015:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202520.449027:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202520.449031:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202520.449041:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1791 00000100:00080000:0.0:1788202520.449053:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202520.449060:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202520.449102:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442880512/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202520.449131:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202520.449134:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202520.449135:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202520.449139:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1791 00000100:00080000:0.0:1788202520.449141:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202520.449144:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202520.449154:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202520.449165:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442880640/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202520.449175:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202520.449177:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202520.449180:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202520.449188:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1791 00000100:00100000:3.0:1788202520.449188:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442880512:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202520.449193:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202520.449200:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202520.449205:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202520.449213:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442880768/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202520.449219:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202520.449222:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202520.449224:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202520.449226:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1791 00000100:00080000:0.0:1788202520.449229:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202520.449233:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202520.449235:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202520.449242:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442880896/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202520.449269:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202520.449303:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202520/real 1788202520] req@ffff9023450b8380 x1875064442880512/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202520.449316:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442880512/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202520.449325:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202520.449327:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202520.449330:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442880512:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202520.449349:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442880640:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202520.449369:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202520.449377:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202520/real 1788202520] req@ffff9023450b8a80 x1875064442880640/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202520.449384:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442880640/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202520.449393:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202520.449395:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202520.449397:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442880640:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202520.449401:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442880768:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202520.449411:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202520.449416:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202520/real 1788202520] req@ffff9023450bb800 x1875064442880768/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202520.449423:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442880768/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202520.449428:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202520.449430:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202520.449431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442880768:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202520.449435:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442880896:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202520.449443:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202520.449448:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202520/real 1788202520] req@ffff9023450b9500 x1875064442880896/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202520.449454:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442880896/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202520.449458:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202520.449460:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202520.449462:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442880896:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202521.281084:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202522.305031:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202523.329100:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202524.353193:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202525.377045:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202525.569006:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202525.569016:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202525.569021:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202525.569032:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1797 00000100:00080000:1.0:1788202525.569044:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202525.569051:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202525.569091:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442881024/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202525.569116:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202525.569119:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202525.569120:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202525.569123:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1797 00000100:00080000:1.0:1788202525.569125:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202525.569128:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202525.569137:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202525.569147:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442881152/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202525.569155:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202525.569158:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202525.569159:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202525.569162:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1797 00000100:00080000:1.0:1788202525.569164:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202525.569166:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202525.569169:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202525.569175:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442881024:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202525.569180:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442881280/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202525.569194:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202525.569197:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202525.569198:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202525.569201:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1797 00000100:00080000:1.0:1788202525.569203:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202525.569205:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202525.569208:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202525.569216:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442881408/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202525.569258:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202525.569306:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202525/real 1788202525] req@ffff90234519ea00 x1875064442881024/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202525.569326:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442881024/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202525.569338:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202525.569343:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202525.569347:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442881024:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202525.569356:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442881152:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202525.569377:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202525.569388:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202525/real 1788202525] req@ffff90234519c380 x1875064442881152/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202525.569401:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442881152/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202525.569410:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202525.569413:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202525.569416:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442881152:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202525.569443:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442881280:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202525.569460:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202525.569471:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202525/real 1788202525] req@ffff90234519f480 x1875064442881280/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202525.569482:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442881280/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202525.569492:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202525.569495:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202525.569498:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442881280:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202525.569504:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442881408:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202525.569520:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202525.569529:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202525/real 1788202525] req@ffff90234519dc00 x1875064442881408/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202525.569558:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442881408/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202525.569566:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202525.569570:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202525.569574:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442881408:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202526.401099:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202527.425003:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202528.449081:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202529.472993:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202530.497045:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202530.688997:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202530.689008:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202530.689012:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202530.689020:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1802 00000100:00080000:2.0:1788202530.689029:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202530.689034:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202530.689065:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442881536/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202530.689086:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202530.689089:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202530.689090:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202530.689093:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1802 00000100:00080000:2.0:1788202530.689095:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202530.689097:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202530.689105:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202530.689113:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442881664/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202530.689121:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202530.689125:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202530.689126:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202530.689126:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442881536:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202530.689133:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1802 00000100:00080000:2.0:1788202530.689137:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202530.689138:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202530.689141:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202530.689150:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442881792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202530.689155:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202530.689158:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202530.689159:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202530.689162:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1802 00000100:00080000:2.0:1788202530.689177:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202530.689179:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202530.689182:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202530.689185:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202530.689199:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442881920/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202530.689218:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202530/real 1788202530] req@ffff90234598f800 x1875064442881536/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202530.689231:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442881536/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202530.689239:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202530.689242:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202530.689245:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442881536:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202530.689261:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442881664:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202530.689274:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202530.689281:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202530/real 1788202530] req@ffff90234598dc00 x1875064442881664/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202530.689289:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442881664/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202530.689294:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202530.689296:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202530.689298:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442881664:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202530.689301:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442881792:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202530.689311:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202530.689316:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202530/real 1788202530] req@ffff90234598c700 x1875064442881792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202530.689322:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442881792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202530.689327:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202530.689328:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202530.689330:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442881792:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202530.689334:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442881920:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202530.689342:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202530.689347:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202530/real 1788202530] req@ffff90234598d180 x1875064442881920/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202530.689353:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442881920/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202530.689357:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202530.689359:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202530.689360:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442881920:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202531.521037:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202532.545063:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202533.569097:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202534.593083:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202535.617060:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202535.809032:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202535.809043:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202535.809047:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202535.809057:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1807 00000100:00080000:0.0:1788202535.809069:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202535.809076:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202535.809129:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442882048/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202535.809162:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202535.809165:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202535.809166:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202535.809170:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1807 00000100:00080000:0.0:1788202535.809173:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202535.809177:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202535.809187:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202535.809200:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442882176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202535.809208:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202535.809211:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202535.809212:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202535.809215:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1807 00000100:00080000:0.0:1788202535.809217:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202535.809220:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202535.809222:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202535.809231:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442882048:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202535.809232:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442882304/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202535.809247:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202535.809250:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202535.809251:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202535.809254:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1807 00000100:00080000:0.0:1788202535.809256:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202535.809258:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202535.809261:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202535.809269:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442882432/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202535.809320:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202535.809398:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202535/real 1788202535] req@ffff9023450b8700 x1875064442882048/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202535.809421:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442882048/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202535.809433:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202535.809438:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202535.809442:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442882048:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202535.809451:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442882176:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202535.809476:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202535.809490:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202535/real 1788202535] req@ffff9023450b9c00 x1875064442882176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202535.809504:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442882176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202535.809514:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202535.809517:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202535.809521:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442882176:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202535.809550:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442882304:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202535.809572:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202535.809583:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202535/real 1788202535] req@ffff9023450b9f80 x1875064442882304/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202535.809597:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442882304/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202535.809607:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202535.809611:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202535.809614:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442882304:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202535.809620:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442882432:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202535.809640:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202535.809651:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202535/real 1788202535] req@ffff9023450bad80 x1875064442882432/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202535.809663:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442882432/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202535.809678:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202535.809681:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202535.809685:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442882432:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202536.641017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202537.665019:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202538.689096:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202539.713057:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202540.737095:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202540.928949:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202540.928957:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202540.928960:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202540.928969:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1812 00000100:00080000:0.0:1788202540.928978:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202540.928984:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202540.929017:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442882560/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202540.929041:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202540.929044:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202540.929045:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202540.929048:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1812 00000100:00080000:0.0:1788202540.929055:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202540.929058:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202540.929064:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202540.929073:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442882688/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202540.929080:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202540.929082:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202540.929084:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202540.929087:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1812 00000100:00080000:0.0:1788202540.929089:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202540.929092:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202540.929095:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202540.929104:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442882560:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202540.929105:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442882816/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202540.929118:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202540.929120:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202540.929122:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202540.929125:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1812 00000100:00080000:0.0:1788202540.929127:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202540.929130:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202540.929133:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202540.929140:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442882944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202540.929171:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202540.929204:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202540/real 1788202540] req@ffff9023450b8000 x1875064442882560/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202540.929218:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442882560/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202540.929226:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202540.929229:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202540.929231:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442882560:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202540.929236:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442882688:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202540.929249:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202540.929256:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202540/real 1788202540] req@ffff9023450ba300 x1875064442882688/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202540.929264:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442882688/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202540.929294:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202540.929296:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202540.929299:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442882688:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202540.929319:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442882816:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202540.929334:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202540.929342:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202540/real 1788202540] req@ffff9023450baa00 x1875064442882816/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202540.929350:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442882816/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202540.929356:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202540.929360:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202540.929363:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442882816:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202540.929369:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442882944:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202540.929388:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202540.929400:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202540/real 1788202540] req@ffff9023450b9180 x1875064442882944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202540.929413:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442882944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202540.929421:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202540.929424:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202540.929426:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442882944:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202541.761085:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202542.785067:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202543.809037:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202544.833065:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202545.857045:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202546.049022:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202546.049037:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202546.049043:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202546.049081:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1817 00000100:00080000:1.0:1788202546.049093:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202546.049101:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202546.049169:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442883072/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202546.049239:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202546.049243:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202546.049245:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202546.049249:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1817 00000100:00080000:1.0:1788202546.049252:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202546.049255:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202546.049265:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202546.049285:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442883200/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202546.049293:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202546.049296:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202546.049298:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202546.049301:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1817 00000100:00080000:1.0:1788202546.049304:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202546.049306:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202546.049309:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202546.049319:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442883328/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202546.049326:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202546.049331:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202546.049335:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202546.049340:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1817 00000100:00100000:3.0:1788202546.049340:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442883072:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202546.049345:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202546.049353:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202546.049364:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202546.049372:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442883456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202546.049435:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202546.049485:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202546/real 1788202546] req@ffff90234519f100 x1875064442883072/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202546.049507:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442883072/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202546.049520:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202546.049525:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202546.049530:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442883072:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202546.049539:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442883200:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202546.049566:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202546.049581:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202546/real 1788202546] req@ffff90234519dc00 x1875064442883200/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202546.049607:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442883200/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202546.049617:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202546.049621:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202546.049625:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442883200:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202546.049653:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442883328:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202546.049677:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202546.049690:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202546/real 1788202546] req@ffff90234519f480 x1875064442883328/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202546.049703:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442883328/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202546.049719:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202546.049722:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202546.049726:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442883328:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202546.049733:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442883456:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202546.049752:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202546.049763:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202546/real 1788202546] req@ffff90234519c380 x1875064442883456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202546.049776:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442883456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202546.049785:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202546.049789:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202546.049792:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442883456:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202546.881084:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202547.905018:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202548.929004:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202549.953012:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202550.977064:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202551.168964:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202551.168973:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202551.168976:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202551.168985:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1822 00000100:00080000:1.0:1788202551.168994:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202551.169000:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202551.169034:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442883584/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202551.169058:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202551.169061:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202551.169063:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202551.169066:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1822 00000100:00080000:1.0:1788202551.169068:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202551.169070:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202551.169077:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202551.169086:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442883712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202551.169111:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202551.169114:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788202551.169119:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442883584:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202551.169123:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202551.169129:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1822 00000100:00080000:1.0:1788202551.169132:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202551.169135:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202551.169140:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202551.169151:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442883840/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202551.169159:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202551.169161:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202551.169162:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202551.169164:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1822 00000100:00080000:1.0:1788202551.169167:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202551.169169:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202551.169172:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202551.169178:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202551.169182:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442883968/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202551.169214:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202551/real 1788202551] req@ffff90234519ea00 x1875064442883584/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202551.169229:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442883584/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202551.169236:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202551.169239:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202551.169243:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442883584:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202551.169262:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442883712:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202551.169280:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202551.169293:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202551/real 1788202551] req@ffff90234519c700 x1875064442883712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202551.169301:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442883712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202551.169307:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202551.169310:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202551.169312:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442883712:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202551.169316:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442883840:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202551.169333:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202551.169341:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202551/real 1788202551] req@ffff90234519fb80 x1875064442883840/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202551.169350:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442883840/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202551.169356:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202551.169359:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202551.169361:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442883840:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202551.169367:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442883968:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202551.169381:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202551.169389:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202551/real 1788202551] req@ffff90234519df80 x1875064442883968/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202551.169397:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442883968/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202551.169403:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202551.169406:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202551.169408:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442883968:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202552.001088:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202553.025086:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202554.049092:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202555.073060:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202556.097134:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202556.289036:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202556.289051:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202556.289058:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202556.289070:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1827 00000100:00080000:1.0:1788202556.289086:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202556.289104:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202556.289156:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442884096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202556.289194:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202556.289203:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202556.289212:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202556.289218:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1827 00000100:00080000:1.0:1788202556.289223:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202556.289228:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202556.289244:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202556.289261:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442884224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202556.289272:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202556.289276:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202556.289278:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202556.289282:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1827 00000100:00080000:1.0:1788202556.289285:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202556.289290:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202556.289295:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202556.289295:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442884096:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202556.289305:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442884352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202556.289315:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202556.289318:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202556.289320:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202556.289323:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1827 00000100:00080000:1.0:1788202556.289327:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202556.289331:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202556.289334:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202556.289343:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442884480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202556.289374:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202556.289423:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202556/real 1788202556] req@ffff90234519ed80 x1875064442884096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202556.289439:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442884096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202556.289448:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202556.289451:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202556.289454:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442884096:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202556.289460:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442884224:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202556.289479:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202556.289488:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202556/real 1788202556] req@ffff90234519d880 x1875064442884224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202556.289496:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442884224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202556.289507:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202556.289509:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202556.289512:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442884224:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202556.289533:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442884352:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202556.289545:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202556.289552:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202556/real 1788202556] req@ffff90234519c000 x1875064442884352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202556.289559:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442884352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202556.289565:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202556.289567:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202556.289569:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442884352:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202556.289572:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442884480:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202556.289583:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202556.289589:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202556/real 1788202556] req@ffff90234519e300 x1875064442884480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202556.289596:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442884480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202556.289601:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202556.289603:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202556.289606:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442884480:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202557.121092:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202558.145092:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202559.169045:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202560.193129:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202561.217073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202561.409050:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202561.409061:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202561.409065:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202561.409076:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1832 00000100:00080000:2.0:1788202561.409089:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202561.409102:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202561.409143:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442884608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202561.409175:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202561.409179:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202561.409181:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202561.409186:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1832 00000100:00080000:2.0:1788202561.409191:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202561.409196:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202561.409206:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202561.409222:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442884736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202561.409234:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202561.409238:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202561.409240:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202561.409245:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1832 00000100:00080000:2.0:1788202561.409249:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202561.409257:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202561.409264:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442884608:192.168.201.130@tcp:250:kworker.0 00000008:00100000:2.0:1788202561.409265:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202561.409286:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442884864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202561.409301:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202561.409305:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202561.409308:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202561.409315:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1832 00000100:00080000:2.0:1788202561.409319:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202561.409324:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202561.409330:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202561.409343:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442884992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202561.409399:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202561.409458:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202561/real 1788202561] req@ffff90234598e300 x1875064442884608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202561.409479:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442884608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202561.409490:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202561.409494:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202561.409497:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442884608:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202561.409505:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442884736:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202561.409527:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202561.409536:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202561/real 1788202561] req@ffff90234598f480 x1875064442884736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202561.409545:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442884736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202561.409550:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202561.409552:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202561.409554:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442884736:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202561.409580:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442884864:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202561.409592:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202561.409599:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202561/real 1788202561] req@ffff90234598c380 x1875064442884864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202561.409607:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442884864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202561.409613:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202561.409615:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202561.409617:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442884864:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202561.409621:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442884992:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202561.409631:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202561.409637:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202561/real 1788202561] req@ffff90234598df80 x1875064442884992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202561.409645:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442884992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202561.409650:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202561.409651:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202561.409653:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442884992:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202562.241077:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202563.265057:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202564.289093:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202565.313023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202566.337066:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202566.528990:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202566.529001:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202566.529006:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202566.529017:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1838 00000100:00080000:2.0:1788202566.529031:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202566.529039:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202566.529096:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442885120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202566.529119:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202566.529122:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202566.529123:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202566.529126:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1838 00000100:00080000:2.0:1788202566.529129:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202566.529132:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202566.529142:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202566.529153:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442885248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202566.529160:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202566.529163:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202566.529164:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202566.529168:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1838 00000100:00080000:2.0:1788202566.529171:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202566.529175:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202566.529178:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202566.529188:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442885120:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202566.529189:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442885376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202566.529215:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202566.529218:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202566.529219:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202566.529223:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1838 00000100:00080000:2.0:1788202566.529227:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202566.529231:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202566.529234:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202566.529244:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442885504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202566.529278:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202566.529316:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202566/real 1788202566] req@ffff90234598fb80 x1875064442885120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202566.529336:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442885120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202566.529347:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202566.529352:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202566.529357:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442885120:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202566.529366:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442885248:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202566.529393:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202566.529421:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202566/real 1788202566] req@ffff90234598ed80 x1875064442885248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202566.529437:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442885248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202566.529447:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202566.529451:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202566.529455:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442885248:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202566.529492:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442885376:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202566.529514:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202566.529526:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202566/real 1788202566] req@ffff90234598f100 x1875064442885376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202566.529534:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442885376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202566.529540:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202566.529542:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202566.529545:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442885376:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202566.529550:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442885504:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202566.529563:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202566.529569:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202566/real 1788202566] req@ffff90234598df80 x1875064442885504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202566.529576:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442885504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202566.529581:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202566.529583:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202566.529585:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442885504:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202567.361085:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202568.385080:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202569.409143:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202570.433073:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202571.457087:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202571.648972:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202571.648984:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202571.648988:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202571.648999:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1843 00000100:00080000:2.0:1788202571.649012:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202571.649019:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202571.649076:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442885632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202571.649107:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202571.649110:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202571.649111:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202571.649114:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1843 00000100:00080000:2.0:1788202571.649116:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202571.649119:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202571.649129:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202571.649139:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442885760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202571.649148:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202571.649150:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202571.649152:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202571.649155:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1843 00000100:00080000:2.0:1788202571.649158:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202571.649161:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202571.649166:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202571.649177:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442885888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202571.649181:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442885632:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202571.649187:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202571.649193:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202571.649195:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202571.649200:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1843 00000100:00080000:2.0:1788202571.649204:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202571.649208:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202571.649213:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202571.649224:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442886016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202571.649303:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442885760:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202571.649349:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442885888:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202571.649379:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442886016:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202572.481034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202573.505020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202574.529062:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788202574.721177:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:1.0:1788202574.721224:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788202574.721243:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788202574.721286:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788202574.721297:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788202574.721305:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788202574.721405:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202571/real 1788202574] req@ffff90234598e300 x1875064442885888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202574.721437:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442885888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202574.721459:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202574.721465:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202574.721469:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442885888:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202574.721482:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202571/real 1788202574] req@ffff90234598d180 x1875064442886016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202574.721497:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442886016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202574.721506:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202574.721510:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202574.721514:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442886016:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202574.721523:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202571/real 1788202574] req@ffff90234598c380 x1875064442885632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202574.721538:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442885632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202574.721547:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202574.721550:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202574.722352:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442885632:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202574.722365:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202571/real 1788202574] req@ffff90234598f480 x1875064442885760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202574.722383:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442885760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202574.722392:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202574.722395:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202574.722399:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442885760:192.168.201.130@tcp:38:kworker.0 00010000:00010000:2.0:1788202575.553033:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202576.577023:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202576.769013:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202576.769021:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202576.769023:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202576.769032:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1848 00000100:00080000:0.0:1788202576.769041:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202576.769047:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202576.769079:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442886144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202576.769102:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202576.769105:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202576.769107:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202576.769117:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1848 00000100:00080000:0.0:1788202576.769123:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202576.769128:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202576.769139:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202576.769154:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442886272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202576.769162:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442886144:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202576.769171:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202576.769176:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202576.769179:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202576.769184:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1848 00000100:00080000:0.0:1788202576.769188:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202576.769192:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202576.769198:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202576.769211:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442886400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202576.769221:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202576.769224:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202576.769226:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202576.769230:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1848 00000800:00000100:3.0:1788202576.769236:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788202576.769238:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202576.769245:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202576.769250:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202576.769261:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442886528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202576.769286:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202576/real 1788202576] req@ffff9023450b9180 x1875064442886144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202576.769309:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442886144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202576.769321:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202576.769326:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202576.769330:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442886144:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202576.769356:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442886272:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202576.769382:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202576.769395:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202576/real 1788202576] req@ffff9023450baa00 x1875064442886272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202576.769409:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442886272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202576.769418:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202576.769421:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202576.769425:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442886272:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202576.769431:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442886400:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202576.769450:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202576.769460:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202576/real 1788202576] req@ffff9023450ba300 x1875064442886400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202576.769474:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442886400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202576.769484:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202576.769488:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202576.769492:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442886400:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202576.769498:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442886528:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202576.769519:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202576.769529:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202576/real 1788202576] req@ffff9023450b8000 x1875064442886528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202576.769541:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442886528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202576.769550:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202576.769554:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202576.769557:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442886528:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202577.601011:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202578.625007:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202579.649035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202580.673048:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202581.697103:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202581.889068:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202581.889082:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202581.889087:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202581.889102:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1853 00000100:00080000:1.0:1788202581.889115:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202581.889124:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202581.889179:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442886656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202581.889227:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202581.889232:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202581.889237:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202581.889243:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1853 00000100:00080000:1.0:1788202581.889247:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202581.889252:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202581.889264:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202581.889281:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442886784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202581.889286:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442886656:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202581.889302:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202581.889307:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202581.889309:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202581.889322:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1853 00000100:00080000:1.0:1788202581.889327:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202581.889332:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202581.889338:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202581.889351:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442886912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202581.889362:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788202581.889365:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788202581.889369:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202581.889373:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202581.889379:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1853 00000100:00080000:1.0:1788202581.889385:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202581.889390:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202581.889395:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202581.889407:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442887040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202581.889415:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202581/real 1788202581] req@ffff90234519f800 x1875064442886656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202581.889436:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442886656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202581.889451:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202581.889454:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202581.889458:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442886656:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202581.889481:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442886784:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202581.889498:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202581.889507:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202581/real 1788202581] req@ffff90234519e300 x1875064442886784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202581.889516:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442886784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202581.889522:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202581.889524:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202581.889526:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442886784:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202581.889530:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442886912:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202581.889542:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202581.889548:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202581/real 1788202581] req@ffff90234519c000 x1875064442886912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202581.889569:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442886912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202581.889577:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202581.889579:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202581.889581:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442886912:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202581.889586:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442887040:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202581.889601:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202581.889609:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202581/real 1788202581] req@ffff90234519d880 x1875064442887040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202581.889617:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442887040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202581.889623:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202581.889625:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202581.889627:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442887040:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202582.721077:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202583.745031:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202584.769008:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202585.793058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202586.817009:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202587.008977:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202587.008985:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202587.008989:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202587.008999:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1858 00000100:00080000:1.0:1788202587.009011:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202587.009017:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202587.009049:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442887168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202587.009076:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202587.009079:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202587.009081:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202587.009084:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1858 00000100:00080000:1.0:1788202587.009087:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202587.009090:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202587.009097:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202587.009109:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442887296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202587.009118:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202587.009123:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202587.009124:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202587.009127:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1858 00000100:00080000:1.0:1788202587.009129:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202587.009132:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202587.009135:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202587.009138:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442887168:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202587.009147:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442887424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202587.009156:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202587.009159:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202587.009160:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202587.009164:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1858 00000100:00080000:1.0:1788202587.009167:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202587.009170:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202587.009174:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202587.009184:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442887552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202587.009215:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202587.009256:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202587/real 1788202587] req@ffff90234519ed80 x1875064442887168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202587.009274:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442887168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202587.009286:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202587.009291:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202587.009296:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442887168:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202587.009326:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442887296:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202587.009386:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202587.009398:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202587/real 1788202587] req@ffff90234519df80 x1875064442887296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202587.009408:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442887296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202587.009417:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202587.009420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202587.009423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442887296:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202587.009429:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442887424:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202587.009442:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202587.009450:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202587/real 1788202587] req@ffff90234519fb80 x1875064442887424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202587.009460:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442887424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202587.009466:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202587.009491:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202587.009494:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442887424:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202587.009498:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442887552:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202587.009513:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202587.009521:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202587/real 1788202587] req@ffff90234519c700 x1875064442887552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202587.009530:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442887552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202587.009535:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202587.009538:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202587.009540:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442887552:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202587.841078:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202588.865107:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202589.889042:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202590.913085:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202591.937020:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202592.128976:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202592.128987:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202592.128992:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202592.129003:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1863 00000100:00080000:0.0:1788202592.129015:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202592.129022:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202592.129062:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442887680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202592.129092:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202592.129097:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202592.129099:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202592.129104:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1863 00000100:00080000:0.0:1788202592.129108:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202592.129114:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202592.129124:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202592.129147:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442887808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202592.129153:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442887680:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202592.129168:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202592.129172:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202592.129174:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202592.129180:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1863 00000100:00080000:0.0:1788202592.129184:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202592.129189:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202592.129194:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202592.129208:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442887936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202592.129225:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788202592.129233:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202592.129238:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202592.129240:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202592.129246:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1863 00000100:00080000:0.0:1788202592.129250:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202592.129255:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202592.129260:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202592.129271:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202592/real 1788202592] req@ffff9023450bad80 x1875064442887680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202592.129290:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442888064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202592.129296:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442887680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202592.129319:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202592.129324:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202592.129329:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442887680:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202592.129354:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442887808:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202592.129378:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202592.129424:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202592/real 1788202592] req@ffff9023450b9f80 x1875064442887808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202592.129438:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442887808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202592.129448:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202592.129452:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202592.129456:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442887808:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202592.129463:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442887936:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202592.129493:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202592.129505:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202592/real 1788202592] req@ffff9023450b9c00 x1875064442887936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202592.129519:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442887936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202592.129529:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202592.129533:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202592.129536:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442887936:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202592.129543:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442888064:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202592.129561:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202592.129573:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202592/real 1788202592] req@ffff9023450b8700 x1875064442888064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202592.129586:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442888064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202592.129602:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202592.129606:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202592.129609:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442888064:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202592.961023:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202593.985105:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202595.009085:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202596.033082:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202597.057139:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202597.249013:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202597.249024:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202597.249033:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202597.249045:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1868 00000100:00080000:0.0:1788202597.249057:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202597.249065:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202597.249120:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442888192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202597.249150:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202597.249153:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202597.249155:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202597.249158:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1868 00000100:00080000:0.0:1788202597.249160:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202597.249163:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202597.249173:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202597.249184:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442888320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202597.249192:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202597.249195:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202597.249196:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202597.249198:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1868 00000100:00080000:0.0:1788202597.249200:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202597.249203:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202597.249206:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202597.249216:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442888192:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202597.249221:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442888448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202597.249232:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202597.249235:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202597.249236:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202597.249239:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1868 00000100:00080000:0.0:1788202597.249242:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202597.249244:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202597.249247:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202597.249255:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442888576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202597.249312:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202597.249363:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202597/real 1788202597] req@ffff9023450b9500 x1875064442888192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202597.249382:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442888192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202597.249390:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202597.249393:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202597.249397:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442888192:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202597.249403:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442888320:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202597.249421:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202597.249429:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202597/real 1788202597] req@ffff9023450bb800 x1875064442888320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202597.249438:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442888320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202597.249444:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202597.249446:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202597.249448:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442888320:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202597.249472:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442888448:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202597.249492:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202597.249504:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202597/real 1788202597] req@ffff9023450b8a80 x1875064442888448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202597.249518:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442888448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202597.249528:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202597.249532:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202597.249535:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442888448:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202597.249539:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442888576:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202597.249555:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202597.249563:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202597/real 1788202597] req@ffff9023450b8380 x1875064442888576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202597.249571:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442888576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202597.249576:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202597.249579:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202597.249581:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442888576:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202598.081063:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202599.105083:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202600.129055:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202601.153074:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202602.177079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202602.369072:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202602.369083:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202602.369087:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202602.369098:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1873 00000100:00080000:2.0:1788202602.369110:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202602.369117:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202602.369167:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442888704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202602.369193:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202602.369195:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202602.369197:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202602.369221:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1873 00000100:00080000:2.0:1788202602.369225:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202602.369228:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202602.369238:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202602.369251:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442888832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202602.369263:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202602.369267:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202602.369268:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202602.369273:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1873 00000100:00080000:2.0:1788202602.369278:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202602.369283:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202602.369290:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202602.369309:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442888960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202602.369315:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442888704:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202602.369323:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202602.369329:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202602.369331:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202602.369335:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1873 00000100:00080000:2.0:1788202602.369338:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202602.369342:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202602.369347:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202602.369360:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442889088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202602.369399:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202602.369452:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202602/real 1788202602] req@ffff90234598c700 x1875064442888704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202602.369473:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442888704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202602.369484:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202602.369488:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202602.369492:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442888704:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202602.369501:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442888832:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202602.369522:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202602.369535:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202602/real 1788202602] req@ffff90234598dc00 x1875064442888832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202602.369547:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442888832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202602.369556:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202602.369560:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202602.369563:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442888832:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202602.369589:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442888960:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202602.369608:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202602.369618:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202602/real 1788202602] req@ffff90234598f800 x1875064442888960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202602.369630:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442888960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202602.369644:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202602.369648:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202602.369651:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442888960:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202602.369657:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442889088:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202602.369673:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202602.369683:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202602/real 1788202602] req@ffff90234598ce00 x1875064442889088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202602.369694:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442889088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202602.369702:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202602.369706:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202602.369709:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442889088:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202603.201062:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202604.225020:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202605.249103:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202606.273129:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202607.297085:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202607.489023:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202607.489032:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202607.489036:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202607.489045:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1879 00000100:00080000:0.0:1788202607.489054:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202607.489059:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202607.489092:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442889216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202607.489138:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202607.489142:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202607.489144:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202607.489149:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1879 00000100:00080000:0.0:1788202607.489153:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202607.489157:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202607.489167:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202607.489183:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442889344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202607.489195:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442889216:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202607.489197:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202607.489207:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202607.489209:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202607.489214:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1879 00000100:00080000:0.0:1788202607.489219:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202607.489223:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202607.489228:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202607.489239:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442889472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202607.489252:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202607.489255:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202607.489264:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788202607.489266:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788202607.489272:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1879 00000100:00080000:0.0:1788202607.489277:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202607.489282:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202607.489288:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202607.489302:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442889600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202607.489322:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202607/real 1788202607] req@ffff9023450ba680 x1875064442889216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202607.489342:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442889216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202607.489352:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202607.489356:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202607.489360:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442889216:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202607.489383:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442889344:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202607.489413:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202607.489425:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202607/real 1788202607] req@ffff9023450b8e00 x1875064442889344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202607.489437:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442889344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202607.489446:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202607.489449:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202607.489453:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442889344:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202607.489459:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442889472:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202607.489478:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202607.489489:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202607/real 1788202607] req@ffff9023450bbb80 x1875064442889472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202607.489500:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442889472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202607.489508:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202607.489520:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202607.489524:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442889472:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202607.489531:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442889600:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202607.489549:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202607.489560:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202607/real 1788202607] req@ffff9023450bb480 x1875064442889600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202607.489572:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442889600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202607.489580:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202607.489583:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202607.489587:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442889600:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202608.321087:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202609.345046:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202610.369100:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202611.393084:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202612.417064:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202612.608989:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202612.609006:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202612.609010:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202612.609020:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1884 00000100:00080000:0.0:1788202612.609059:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202612.609067:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202612.609107:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442889728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202612.609146:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202612.609151:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202612.609153:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202612.609157:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1884 00000100:00080000:0.0:1788202612.609161:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202612.609165:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202612.609175:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202612.609189:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442889856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202612.609194:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442889728:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202612.609210:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202612.609214:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202612.609216:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202612.609222:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1884 00000100:00080000:0.0:1788202612.609226:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202612.609230:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202612.609236:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202612.609248:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442889984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202612.609258:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788202612.609259:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202612.609265:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202612.609267:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202612.609273:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1884 00000100:00080000:0.0:1788202612.609278:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202612.609282:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202612.609288:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202612.609294:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202612/real 1788202612] req@ffff9023450bb480 x1875064442889728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202612.609304:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442890112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202612.609313:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442889728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202612.609325:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202612.609329:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202612.609332:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442889728:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202612.609352:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442889856:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202612.609366:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202612.609374:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202612/real 1788202612] req@ffff9023450bbb80 x1875064442889856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202612.609382:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442889856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202612.609392:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202612.609394:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202612.609396:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442889856:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202612.609400:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442889984:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202612.609410:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202612.609416:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202612/real 1788202612] req@ffff9023450b8e00 x1875064442889984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202612.609423:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442889984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202612.609429:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202612.609430:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202612.609432:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442889984:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202612.609436:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442890112:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202612.609445:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202612.609450:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202612/real 1788202612] req@ffff9023450ba680 x1875064442890112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202612.609456:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442890112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202612.609461:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202612.609463:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202612.609465:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442890112:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202613.441040:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202614.465060:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202615.489062:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202616.513047:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202617.537067:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202617.728951:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202617.728959:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202617.728963:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202617.728971:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1889 00000100:00080000:0.0:1788202617.728981:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202617.728986:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202617.729017:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442890240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202617.729045:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202617.729048:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202617.729050:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202617.729053:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1889 00000100:00080000:0.0:1788202617.729055:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202617.729057:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202617.729064:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202617.729074:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442890368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202617.729082:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202617.729085:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202617.729086:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202617.729089:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442890240:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202617.729091:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1889 00000100:00080000:0.0:1788202617.729096:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202617.729101:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202617.729104:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202617.729114:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442890496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202617.729130:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202617.729134:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202617.729136:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202617.729141:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1889 00000100:00080000:0.0:1788202617.729146:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788202617.729152:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788202617.729153:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202617.729162:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202617.729177:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442890624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202617.729201:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202617/real 1788202617] req@ffff9023450b8380 x1875064442890240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202617.729217:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442890240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202617.729230:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202617.729233:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202617.729236:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442890240:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202617.729255:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442890368:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202617.729274:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202617.729288:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202617/real 1788202617] req@ffff9023450b8a80 x1875064442890368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202617.729302:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442890368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202617.729311:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202617.729314:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202617.729318:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442890368:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202617.729324:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442890496:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202617.729340:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202617.729348:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202617/real 1788202617] req@ffff9023450bb800 x1875064442890496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202617.729356:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442890496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202617.729362:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202617.729364:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202617.729366:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442890496:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202617.729369:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442890624:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202617.729378:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202617.729384:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202617/real 1788202617] req@ffff9023450b9500 x1875064442890624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202617.729391:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442890624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202617.729396:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202617.729398:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202617.729400:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442890624:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202618.561056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202619.585042:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202620.609110:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202621.633054:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202622.657069:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202622.849003:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202622.849015:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202622.849019:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202622.849030:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1894 00000100:00080000:0.0:1788202622.849043:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202622.849050:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202622.849090:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442890752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202622.849130:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202622.849135:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202622.849137:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202622.849142:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1894 00000100:00080000:0.0:1788202622.849145:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202622.849150:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202622.849160:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202622.849178:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442890880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202622.849188:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442890752:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202622.849192:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202622.849198:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202622.849200:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202622.849205:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1894 00000100:00080000:0.0:1788202622.849209:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202622.849215:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202622.849221:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202622.849237:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442891008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202622.849248:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788202622.849253:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788202622.849255:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202622.849259:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202622.849268:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1894 00000100:00080000:0.0:1788202622.849273:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202622.849279:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202622.849286:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202622.849298:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202622/real 1788202622] req@ffff9023450b8700 x1875064442890752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202622.849303:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442891136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202622.849338:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442890752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202622.849348:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202622.849350:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202622.849357:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442890752:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202622.849378:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442890880:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202622.849399:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202622.849423:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202622/real 1788202622] req@ffff9023450b9c00 x1875064442890880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202622.849440:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442890880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202622.849448:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202622.849451:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202622.849456:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442890880:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202622.849464:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442891008:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202622.849478:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202622.849488:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202622/real 1788202622] req@ffff9023450b9f80 x1875064442891008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202622.849509:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442891008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202622.849516:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202622.849520:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202622.849525:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442891008:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202622.849531:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442891136:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202622.849543:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202622.849554:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202622/real 1788202622] req@ffff9023450bad80 x1875064442891136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202622.849561:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442891136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202622.849567:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202622.849570:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202622.849572:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442891136:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202623.681022:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202624.705027:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202625.729004:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202626.753069:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202627.777054:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202627.968976:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202627.968985:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202627.968988:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202627.968996:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1899 00000100:00080000:0.0:1788202627.969005:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202627.969010:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202627.969054:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442891264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202627.969083:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202627.969087:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202627.969089:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202627.969094:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1899 00000100:00080000:0.0:1788202627.969098:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202627.969103:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202627.969113:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202627.969127:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442891392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202627.969135:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442891264:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202627.969143:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202627.969148:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202627.969151:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202627.969156:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1899 00000100:00080000:0.0:1788202627.969161:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202627.969165:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202627.969171:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202627.969184:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442891520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202627.969196:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202627.969199:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202627.969201:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202627.969208:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1899 00000800:00000100:2.0:1788202627.969209:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788202627.969216:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202627.969222:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202627.969228:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202627.969242:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442891648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202627.969264:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202627/real 1788202627] req@ffff9023450b8000 x1875064442891264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202627.969292:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442891264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202627.969302:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202627.969307:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202627.969311:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442891264:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202627.969333:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442891392:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202627.969356:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202627.969369:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202627/real 1788202627] req@ffff9023450ba300 x1875064442891392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202627.969381:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442891392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202627.969389:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202627.969392:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202627.969396:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442891392:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202627.969402:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442891520:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202627.969419:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202627.969429:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202627/real 1788202627] req@ffff9023450baa00 x1875064442891520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202627.969441:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442891520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202627.969455:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202627.969458:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202627.969462:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442891520:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202627.969468:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442891648:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202627.969485:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202627.969494:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202627/real 1788202627] req@ffff9023450b9180 x1875064442891648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202627.969506:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442891648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202627.969513:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202627.969517:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202627.969520:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442891648:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202628.801024:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202629.825043:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202630.849055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202631.873080:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202632.897078:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202633.089021:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202633.089037:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202633.089042:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202633.089056:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1904 00000100:00080000:1.0:1788202633.089071:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202633.089080:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202633.089133:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442891776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202633.089186:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202633.089190:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202633.089193:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202633.089198:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1904 00000100:00080000:1.0:1788202633.089202:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202633.089207:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202633.089220:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202633.089237:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442891904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202633.089244:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442891776:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202633.089258:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202633.089263:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202633.089266:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202633.089271:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1904 00000100:00080000:1.0:1788202633.089276:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202633.089281:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202633.089287:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202633.089302:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442892032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202633.089315:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202633.089319:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202633.089322:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788202633.089326:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788202633.089331:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1904 00000100:00080000:1.0:1788202633.089337:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202633.089343:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202633.089349:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202633.089365:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442892160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202633.089377:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202633/real 1788202633] req@ffff90234519ea00 x1875064442891776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202633.089395:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442891776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202633.089409:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202633.089413:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202633.089417:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442891776:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202633.089441:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442891904:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202633.089457:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202633.089466:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202633/real 1788202633] req@ffff90234519c380 x1875064442891904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202633.089474:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442891904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202633.089480:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202633.089482:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202633.089484:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442891904:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202633.089487:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442892032:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202633.089499:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202633.089505:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202633/real 1788202633] req@ffff90234519f480 x1875064442892032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202633.089512:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442892032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202633.089518:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202633.089519:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202633.089521:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442892032:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202633.089525:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442892160:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202633.089534:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202633.089540:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202633/real 1788202633] req@ffff90234519dc00 x1875064442892160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202633.089547:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442892160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202633.089552:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202633.089554:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202633.089556:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442892160:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202633.921036:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202634.944991:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202635.969076:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202636.993007:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202638.017079:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202638.209043:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202638.209054:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202638.209059:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202638.209069:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1909 00000100:00080000:0.0:1788202638.209082:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202638.209089:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202638.209135:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442892288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202638.209163:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202638.209166:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202638.209167:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202638.209170:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1909 00000100:00080000:0.0:1788202638.209173:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202638.209176:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202638.209187:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202638.209205:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442892416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202638.209213:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202638.209216:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202638.209218:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202638.209221:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1909 00000100:00080000:0.0:1788202638.209223:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202638.209229:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202638.209231:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442892288:192.168.201.130@tcp:250:kworker.0 00000008:00100000:0.0:1788202638.209233:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202638.209246:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442892544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202638.209261:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202638.209265:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202638.209266:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202638.209269:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1909 00000100:00080000:0.0:1788202638.209272:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202638.209275:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202638.209278:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202638.209286:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442892672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202638.209354:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442892416:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202638.209403:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442892544:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202638.209424:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442892672:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202639.041047:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202640.065060:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202641.089026:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202641.281245:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:3.0:1788202641.281318:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202641.281341:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:3.0:1788202641.281400:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:3.0:1788202641.281414:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:3.0:1788202641.281428:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:2.0:1788202641.281499:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202638/real 1788202641] req@ffff9023450ba300 x1875064442892544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202641.281567:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442892544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202641.281583:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202641.281589:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202641.281593:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442892544:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202641.281603:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202638/real 1788202641] req@ffff9023450b8000 x1875064442892672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202641.281617:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442892672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202641.281625:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202641.281629:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202641.281633:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442892672:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202641.281641:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202638/real 1788202641] req@ffff9023450b9180 x1875064442892288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202641.281654:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442892288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202641.281663:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202641.281667:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202641.281671:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442892288:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202641.281679:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202638/real 1788202641] req@ffff9023450baa00 x1875064442892416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202641.281692:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442892416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202641.281701:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202641.281705:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202641.281709:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442892416:192.168.201.130@tcp:38:kworker.0 00010000:00010000:2.0:1788202642.113079:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202643.137057:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202643.328985:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202643.328993:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202643.328997:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202643.329005:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1914 00000100:00080000:0.0:1788202643.329021:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202643.329031:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202643.329066:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442892800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202643.329088:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202643.329090:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202643.329092:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202643.329095:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1914 00000100:00080000:0.0:1788202643.329097:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202643.329099:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202643.329106:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202643.329113:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442892928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202643.329121:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202643.329125:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202643.329127:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202643.329133:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1914 00000100:00080000:0.0:1788202643.329137:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202643.329142:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202643.329148:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202643.329160:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442893056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202643.329173:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442892800:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202643.329197:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202643.329202:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202643.329204:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202643.329210:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1914 00000100:00080000:0.0:1788202643.329215:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202643.329219:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202643.329225:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202643.329242:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442893184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202643.329296:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202643.329351:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202643/real 1788202643] req@ffff9023450bad80 x1875064442892800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202643.329374:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442892800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202643.329388:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202643.329393:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202643.329398:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442892800:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202643.329408:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442892928:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202643.329436:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202643.329452:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202643/real 1788202643] req@ffff9023450b9f80 x1875064442892928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202643.329466:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442892928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202643.329476:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202643.329480:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202643.329483:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442892928:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202643.329539:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442893056:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202643.329561:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202643.329571:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202643/real 1788202643] req@ffff9023450b9c00 x1875064442893056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202643.329579:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442893056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202643.329587:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202643.329590:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202643.329594:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442893056:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202643.329600:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442893184:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202643.329623:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202643.329635:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202643/real 1788202643] req@ffff9023450b8700 x1875064442893184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202643.329647:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442893184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202643.329655:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202643.329659:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202643.329662:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442893184:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202644.161432:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202645.185062:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202646.209091:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202647.233075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202648.257054:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202648.448953:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202648.448964:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202648.448968:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202648.448979:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1919 00000100:00080000:0.0:1788202648.448990:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202648.448998:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202648.449036:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442893312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202648.449058:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202648.449061:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202648.449062:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202648.449065:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1919 00000100:00080000:0.0:1788202648.449068:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202648.449071:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202648.449079:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202648.449088:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442893440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202648.449096:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202648.449099:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202648.449101:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202648.449101:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442893312:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202648.449106:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1919 00000100:00080000:0.0:1788202648.449111:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202648.449114:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202648.449117:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202648.449124:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442893568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202648.449131:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202648.449133:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202648.449135:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202648.449137:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1919 00000100:00080000:0.0:1788202648.449139:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202648.449142:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202648.449144:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202648.449151:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442893696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202648.449174:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202648.449231:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202648/real 1788202648] req@ffff9023450b9500 x1875064442893312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202648.449247:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442893312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202648.449255:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202648.449259:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202648.449263:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442893312:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202648.449282:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442893440:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202648.449299:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202648.449308:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202648/real 1788202648] req@ffff9023450bb800 x1875064442893440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202648.449320:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442893440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202648.449329:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202648.449333:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202648.449337:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442893440:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202648.449352:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442893568:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202648.449374:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202648.449389:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202648/real 1788202648] req@ffff9023450b8a80 x1875064442893568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202648.449447:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442893568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202648.449459:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202648.449462:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202648.449466:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442893568:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202648.449471:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442893696:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202648.449494:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202648.449505:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202648/real 1788202648] req@ffff9023450b8380 x1875064442893696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202648.449514:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442893696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202648.449521:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202648.449523:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202648.449525:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442893696:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202649.281050:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202650.305028:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202651.329048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202652.353050:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202653.377099:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202653.569002:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202653.569010:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202653.569014:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202653.569023:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1925 00000100:00080000:2.0:1788202653.569032:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202653.569038:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202653.569071:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442893824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202653.569096:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202653.569099:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202653.569100:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202653.569103:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1925 00000100:00080000:2.0:1788202653.569106:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202653.569108:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202653.569115:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202653.569124:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064442893952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202653.569133:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202653.569137:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788202653.569137:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442893824:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202653.569141:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202653.569148:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1925 00000100:00080000:2.0:1788202653.569151:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202653.569154:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202653.569157:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202653.569167:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064442894080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202653.569179:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202653.569182:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202653.569184:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202653.569186:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1925 00000100:00080000:2.0:1788202653.569195:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202653.569197:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202653.569200:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202653.569207:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202653.569211:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442894208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202653.569251:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202653/real 1788202653] req@ffff90234598d880 x1875064442893824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202653.569265:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442893824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202653.569273:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202653.569276:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202653.569279:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442893824:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202653.569297:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442893952:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202653.569337:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202653.569348:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202653/real 1788202653] req@ffff90234598e680 x1875064442893952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202653.569357:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064442893952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202653.569362:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202653.569365:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202653.569367:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442893952:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202653.569371:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442894080:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202653.569385:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202653.569392:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202653/real 1788202653] req@ffff90234598ca80 x1875064442894080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202653.569404:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064442894080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202653.569409:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202653.569411:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202653.569413:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442894080:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202653.569418:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442894208:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202653.569434:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202653.569446:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202653/real 1788202653] req@ffff90234598ce00 x1875064442894208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202653.569459:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442894208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202653.569468:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202653.569472:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202653.569475:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442894208:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202654.401098:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202655.424990:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202656.449010:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202657.473140:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202658.497110:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202658.688983:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202658.688992:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202658.688997:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202658.689007:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1930 00000100:00080000:2.0:1788202658.689018:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202658.689025:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202658.689061:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442894336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202658.689090:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202658.689093:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202658.689096:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202658.689101:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1930 00000100:00080000:2.0:1788202658.689106:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202658.689112:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202658.689124:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202658.689142:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442894464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202658.689148:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442894336:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202658.689180:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202658.689184:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202658.689186:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202658.689191:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1930 00000100:00080000:2.0:1788202658.689195:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202658.689200:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202658.689206:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202658.689220:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442894592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202658.689248:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202658.689254:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202658.689258:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202658.689264:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1930 00000100:00080000:2.0:1788202658.689269:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202658.689275:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:3.0:1788202658.689281:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:2.0:1788202658.689287:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202658.689311:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442894720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202658.689330:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202658/real 1788202658] req@ffff90234598f800 x1875064442894336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202658.689349:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442894336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202658.689374:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202658.689380:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202658.689385:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442894336:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202658.689413:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442894464:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202658.689450:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202658.689467:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202658/real 1788202658] req@ffff90234598dc00 x1875064442894464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202658.689481:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442894464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202658.689490:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202658.689494:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202658.689498:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442894464:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202658.689506:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442894592:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202658.689531:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202658.689545:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202658/real 1788202658] req@ffff90234598c700 x1875064442894592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202658.689556:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442894592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202658.689564:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202658.689567:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202658.689571:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442894592:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202658.689577:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442894720:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202658.689595:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202658.689604:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202658/real 1788202658] req@ffff90234598f480 x1875064442894720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202658.689615:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442894720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202658.689623:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202658.689626:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202658.689630:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442894720:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202659.521060:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202660.545034:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202661.569070:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202662.593024:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202663.617062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202663.809006:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202663.809029:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202663.809035:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202663.809049:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1935 00000100:00080000:2.0:1788202663.809068:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202663.809078:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202663.809127:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442894848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202663.809185:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202663.809188:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202663.809189:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202663.809193:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1935 00000100:00080000:2.0:1788202663.809196:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202663.809200:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202663.809209:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202663.809221:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442894976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202663.809229:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202663.809232:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202663.809234:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202663.809239:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1935 00000100:00100000:3.0:1788202663.809241:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442894848:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202663.809242:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202663.809247:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202663.809254:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202663.809260:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442895104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202663.809266:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202663.809269:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202663.809270:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202663.809273:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1935 00000100:00080000:2.0:1788202663.809275:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202663.809277:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202663.809280:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202663.809288:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442895232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202663.809334:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202663.809379:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202663/real 1788202663] req@ffff90234598c380 x1875064442894848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202663.809397:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442894848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202663.809406:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202663.809409:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202663.809413:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442894848:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202663.809439:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442894976:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202663.809457:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202663.809466:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202663/real 1788202663] req@ffff90234598d180 x1875064442894976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202663.809475:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442894976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202663.809481:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202663.809483:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202663.809485:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442894976:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202663.809489:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442895104:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202663.809501:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202663.809507:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202663/real 1788202663] req@ffff90234598e300 x1875064442895104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202663.809515:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442895104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202663.809521:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202663.809523:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202663.809525:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442895104:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202663.809528:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442895232:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202663.809539:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202663.809544:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202663/real 1788202663] req@ffff90234598df80 x1875064442895232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202663.809552:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442895232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202663.809562:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202663.809564:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202663.809566:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442895232:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202664.641076:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202665.665041:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202666.688997:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202667.713004:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202668.737047:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202668.929057:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202668.929066:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202668.929069:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202668.929078:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1940 00000100:00080000:1.0:1788202668.929087:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202668.929093:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202668.929128:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442895360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202668.929156:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202668.929160:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202668.929162:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202668.929174:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1940 00000100:00080000:1.0:1788202668.929179:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202668.929184:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202668.929194:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202668.929200:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442895360:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202668.929211:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442895488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202668.929220:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202668.929223:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202668.929225:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202668.929228:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1940 00000100:00080000:1.0:1788202668.929231:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202668.929234:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202668.929237:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202668.929244:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442895616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202668.929251:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202668.929254:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202668.929255:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202668.929259:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1940 00000100:00080000:1.0:1788202668.929263:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788202668.929268:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788202668.929281:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202668.929287:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202668.929302:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442895744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202668.929307:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202668/real 1788202668] req@ffff90234519f100 x1875064442895360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202668.929336:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442895360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202668.929345:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202668.929348:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202668.929352:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442895360:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202668.929378:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442895488:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202668.929394:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202668.929403:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202668/real 1788202668] req@ffff90234519dc00 x1875064442895488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202668.929411:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442895488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202668.929417:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202668.929419:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202668.929422:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442895488:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202668.929426:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442895616:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202668.929437:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202668.929443:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202668/real 1788202668] req@ffff90234519f480 x1875064442895616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202668.929451:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442895616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202668.929456:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202668.929458:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202668.929460:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442895616:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202668.929463:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442895744:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202668.929473:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202668.929478:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202668/real 1788202668] req@ffff90234519c380 x1875064442895744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202668.929486:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442895744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202668.929490:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202668.929492:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202668.929494:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442895744:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202669.761080:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202670.785060:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202671.809039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202672.833033:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202673.857089:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202674.049010:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202674.049022:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202674.049027:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202674.049042:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1945 00000100:00080000:1.0:1788202674.049057:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202674.049066:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202674.049117:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442895872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202674.049149:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202674.049153:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202674.049155:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202674.049160:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1945 00000100:00080000:1.0:1788202674.049165:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202674.049169:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202674.049181:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202674.049196:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442896000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202674.049207:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202674.049211:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202674.049213:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202674.049213:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442895872:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202674.049217:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1945 00000100:00080000:1.0:1788202674.049220:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202674.049224:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202674.049228:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202674.049241:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442896128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202674.049249:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202674.049252:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202674.049254:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202674.049257:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1945 00000100:00080000:1.0:1788202674.049260:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202674.049264:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202674.049268:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202674.049290:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442896256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202674.049290:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202674.049349:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202674/real 1788202674] req@ffff90234519ea00 x1875064442895872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202674.049367:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442895872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202674.049377:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202674.049380:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202674.049383:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442895872:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202674.049412:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442896000:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202674.049439:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202674.049454:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202674/real 1788202674] req@ffff90234519c700 x1875064442896000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202674.049464:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442896000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202674.049471:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202674.049473:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202674.049475:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442896000:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202674.049480:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442896128:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202674.049495:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202674.049503:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202674/real 1788202674] req@ffff90234519fb80 x1875064442896128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202674.049511:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442896128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202674.049520:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202674.049522:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202674.049524:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442896128:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202674.049527:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442896256:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202674.049537:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202674.049543:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202674/real 1788202674] req@ffff90234519df80 x1875064442896256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202674.049551:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442896256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202674.049559:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202674.049562:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202674.049565:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442896256:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202674.881103:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202675.905103:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202676.929081:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202677.953045:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202678.977036:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202679.169015:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202679.169026:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202679.169031:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202679.169040:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1950 00000100:00080000:0.0:1788202679.169053:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202679.169059:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202679.169097:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442896384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202679.169123:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202679.169126:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202679.169128:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202679.169135:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1950 00000100:00080000:0.0:1788202679.169138:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202679.169141:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202679.169149:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202679.169158:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442896512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202679.169167:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788202679.169170:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442896384:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202679.169171:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202679.169174:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202679.169182:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1950 00000100:00080000:0.0:1788202679.169184:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202679.169186:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202679.169190:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202679.169199:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442896640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202679.169205:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202679.169208:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202679.169209:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202679.169211:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1950 00000100:00080000:0.0:1788202679.169213:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202679.169216:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202679.169218:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202679.169225:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442896768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202679.169248:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202679.169300:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202679/real 1788202679] req@ffff9023450ba680 x1875064442896384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202679.169315:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442896384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202679.169327:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202679.169331:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202679.169334:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442896384:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202679.169356:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442896512:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202679.169381:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202679.169395:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202679/real 1788202679] req@ffff9023450b8e00 x1875064442896512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202679.169408:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442896512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202679.169416:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202679.169420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202679.169423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442896512:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202679.169430:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442896640:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202679.169450:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202679.169461:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202679/real 1788202679] req@ffff9023450bbb80 x1875064442896640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202679.169474:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442896640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202679.169484:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202679.169487:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202679.169491:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442896640:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202679.169497:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442896768:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202679.169516:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202679.169526:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202679/real 1788202679] req@ffff9023450bb480 x1875064442896768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202679.169539:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442896768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202679.169548:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202679.169551:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202679.169555:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442896768:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202680.001018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202681.025013:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202682.049053:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202683.073112:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202684.097158:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202684.289044:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202684.289054:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202684.289057:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202684.289067:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1955 00000100:00080000:2.0:1788202684.289076:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202684.289082:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202684.289115:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442896896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202684.289143:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202684.289146:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202684.289148:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202684.289152:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1955 00000100:00080000:2.0:1788202684.289156:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202684.289160:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202684.289169:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202684.289185:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442897024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202684.289191:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442896896:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202684.289206:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202684.289211:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202684.289214:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202684.289220:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1955 00000100:00080000:2.0:1788202684.289225:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202684.289230:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202684.289236:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202684.289251:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202684.289254:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442897152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202684.289265:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202684.289268:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202684.289269:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202684.289273:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1955 00000100:00080000:2.0:1788202684.289276:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202684.289279:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202684.289284:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202684.289289:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202684/real 1788202684] req@ffff90234598f100 x1875064442896896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202684.289295:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442897280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202684.289320:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442896896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202684.289330:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202684.289333:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202684.289336:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442896896:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202684.289363:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442897024:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202684.289387:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202684.289399:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202684/real 1788202684] req@ffff90234598ed80 x1875064442897024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202684.289408:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442897024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202684.289418:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202684.289420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202684.289423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442897024:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202684.289427:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442897152:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202684.289441:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202684.289448:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202684/real 1788202684] req@ffff90234598fb80 x1875064442897152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202684.289455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442897152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202684.289461:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202684.289462:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202684.289464:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442897152:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202684.289468:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442897280:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202684.289478:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202684.289484:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202684/real 1788202684] req@ffff90234598df80 x1875064442897280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202684.289491:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442897280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202684.289496:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202684.289498:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202684.289501:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442897280:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202685.121061:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202686.145052:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202687.169090:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202688.193043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202689.217072:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202689.408995:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202689.409007:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202689.409011:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202689.409031:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1960 00000100:00080000:2.0:1788202689.409041:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202689.409047:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202689.409087:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442897408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202689.409117:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202689.409121:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202689.409124:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202689.409129:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1960 00000100:00080000:2.0:1788202689.409135:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202689.409141:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202689.409151:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202689.409163:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442897408:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202689.409169:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442897536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202689.409221:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788202689.409236:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202689.409241:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788202689.409258:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202689/real 1788202689] req@ffff90234598e300 x1875064442897408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202689.409268:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202689.409273:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442897408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202689.409279:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1960 00000100:00080000:3.0:1788202689.409285:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202689.409288:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202689.409291:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00080000:2.0:1788202689.409299:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202689.409300:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442897408:192.168.201.130@tcp:250:kworker.0 00000008:00100000:2.0:1788202689.409310:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202689.409325:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442897664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202689.409337:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202689.409343:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788202689.409343:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442897536:192.168.201.130@tcp:38:kworker.0 00000100:00080000:2.0:1788202689.409347:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202689.409354:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1960 00000100:00080000:2.0:1788202689.409359:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202689.409364:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202689.409371:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202689.409377:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202689.409388:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442897792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202689.409396:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202689/real 1788202689] req@ffff90234598d180 x1875064442897536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202689.409423:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442897536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202689.409431:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202689.409436:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202689.409441:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442897536:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202689.409448:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442897664:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202689.409472:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202689.409485:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202689/real 1788202689] req@ffff90234598c380 x1875064442897664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202689.409496:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442897664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202689.409502:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202689.409504:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202689.409506:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442897664:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202689.409518:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442897792:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202689.409532:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202689.409539:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202689/real 1788202689] req@ffff90234598f480 x1875064442897792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202689.409547:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442897792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202689.409552:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202689.409554:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202689.409556:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442897792:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202690.241046:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202691.265018:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202692.289048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202693.312966:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202694.337049:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202694.529011:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202694.529022:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202694.529026:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202694.529037:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1966 00000100:00080000:0.0:1788202694.529049:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202694.529058:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202694.529098:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442897920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202694.529145:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202694.529151:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202694.529153:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202694.529159:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1966 00000100:00080000:0.0:1788202694.529164:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202694.529170:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202694.529182:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202694.529189:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442897920:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202694.529203:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442898048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202694.529218:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202694.529222:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202694.529225:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202694.529230:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1966 00000100:00080000:0.0:1788202694.529235:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202694.529240:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202694.529246:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788202694.529251:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202694.529262:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442898176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202694.529274:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202694.529279:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202694.529281:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202694.529284:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202694/real 1788202694] req@ffff9023450bb480 x1875064442897920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202694.529289:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1966 00000100:00080000:0.0:1788202694.529298:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202694.529304:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202694.529308:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442897920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788202694.529314:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1788202694.529318:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202694.529322:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202694.529327:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442897920:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202694.529332:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442898304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202694.529351:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442898048:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202694.529369:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202694.529379:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202694/real 1788202694] req@ffff9023450bbb80 x1875064442898048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202694.529387:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442898048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202694.529394:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202694.529397:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202694.529401:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442898048:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202694.529407:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442898176:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202694.529431:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202694.529444:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202694/real 1788202694] req@ffff9023450b8e00 x1875064442898176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202694.529456:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442898176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202694.529465:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202694.529468:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202694.529472:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442898176:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202694.529478:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442898304:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202694.529495:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202694.529503:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202694/real 1788202694] req@ffff9023450ba680 x1875064442898304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202694.529511:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442898304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202694.529516:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202694.529518:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202694.529520:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442898304:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202695.361020:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202696.385037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202697.408992:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202698.433114:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202699.457011:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202699.648993:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202699.649003:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202699.649007:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202699.649018:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1971 00000100:00080000:3.0:1788202699.649029:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202699.649037:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202699.649073:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442898432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202699.649100:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202699.649104:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202699.649106:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202699.649110:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1971 00000100:00080000:3.0:1788202699.649114:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202699.649119:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202699.649130:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202699.649150:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442898432:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202699.649150:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442898560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202699.649168:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202699.649171:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202699.649173:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202699.649176:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1971 00000100:00080000:3.0:1788202699.649179:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202699.649182:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202699.649187:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202699.649198:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442898688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202699.649205:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202699.649207:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202699.649209:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202699.649212:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1971 00000100:00080000:3.0:1788202699.649215:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202699.649217:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202699.649220:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788202699.649223:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202699.649230:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442898816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202699.649269:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202699/real 1788202699] req@ffff90235268b100 x1875064442898432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202699.649288:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442898432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202699.649299:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202699.649303:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202699.649307:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442898432:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202699.649330:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442898560:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202699.649350:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202699.649361:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202699/real 1788202699] req@ffff90235268b800 x1875064442898560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202699.649375:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442898560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202699.649384:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202699.649387:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202699.649391:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442898560:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202699.649397:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442898688:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202699.649413:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202699.649423:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202699/real 1788202699] req@ffff902352688000 x1875064442898688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202699.649435:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442898688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202699.649451:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202699.649454:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202699.649457:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442898688:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202699.649463:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442898816:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202699.649478:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202699.649487:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202699/real 1788202699] req@ffff90235268a300 x1875064442898816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202699.649499:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442898816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202699.649508:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202699.649511:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202699.649541:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442898816:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202700.481051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202701.505026:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202702.529069:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202703.553050:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202704.577062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202704.768972:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202704.768980:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202704.768984:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202704.768992:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1976 00000100:00080000:2.0:1788202704.769002:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202704.769007:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202704.769040:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442898944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202704.769063:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202704.769066:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202704.769067:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202704.769070:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1976 00000100:00080000:2.0:1788202704.769072:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202704.769074:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202704.769082:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202704.769094:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442899072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202704.769107:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788202704.769108:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442898944:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202704.769116:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202704.769120:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202704.769126:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1976 00000100:00080000:2.0:1788202704.769130:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202704.769134:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202704.769140:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202704.769152:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442899200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202704.769159:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202704.769162:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202704.769163:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202704.769165:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1976 00000100:00080000:2.0:1788202704.769169:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202704.769172:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202704.769175:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202704.769182:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442899328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202704.769214:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442899072:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202704.769240:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442899200:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202704.769256:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442899328:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202705.601008:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202706.625029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202707.649244:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788202707.841171:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:0.0:1788202707.841213:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788202707.841228:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788202707.841278:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788202707.841288:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788202707.841296:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788202707.841341:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202704/real 1788202707] req@ffff90234598dc00 x1875064442899072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202707.841363:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442899072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202707.841406:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202707.841410:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202707.841414:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442899072:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202707.841421:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202704/real 1788202707] req@ffff90234598f800 x1875064442899200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202707.841434:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442899200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202707.841442:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202707.841446:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202707.841450:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442899200:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202707.841456:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202704/real 1788202707] req@ffff90234598ce00 x1875064442899328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202707.841466:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442899328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202707.841474:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202707.841477:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202707.841481:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442899328:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202707.841487:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202704/real 1788202707] req@ffff90234598c700 x1875064442898944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202707.841499:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442898944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202707.841506:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202707.841508:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202707.841510:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442898944:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788202708.673088:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202709.697023:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202709.889001:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202709.889008:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202709.889011:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202709.889020:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1981 00000100:00080000:3.0:1788202709.889029:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202709.889036:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202709.889067:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442899456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202709.889102:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202709.889105:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202709.889107:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202709.889111:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1981 00000100:00080000:3.0:1788202709.889115:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202709.889127:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202709.889139:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202709.889152:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442899584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202709.889161:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202709.889164:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202709.889166:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202709.889169:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1981 00000100:00080000:3.0:1788202709.889172:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202709.889176:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202709.889180:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202709.889190:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442899712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202709.889199:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202709.889201:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202709.889203:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202709.889206:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1981 00000100:00080000:3.0:1788202709.889209:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202709.889213:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202709.889217:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202709.889226:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442899840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202709.889321:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442899456:192.168.201.130@tcp:250:kworker.0 00000800:00000100:2.0:1788202709.889395:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202709.889434:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202709/real 1788202709] req@ffff902352688e00 x1875064442899456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202709.889450:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442899456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202709.889465:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202709.889469:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202709.889472:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442899456:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202709.889479:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442899584:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202709.889501:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202709.889512:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202709/real 1788202709] req@ffff902352689c00 x1875064442899584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202709.889523:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442899584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202709.889530:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202709.889532:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202709.889535:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442899584:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202709.889541:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442899712:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202709.889558:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202709.889568:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202709/real 1788202709] req@ffff90235268a680 x1875064442899712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202709.889579:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442899712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202709.889586:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202709.889588:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202709.889591:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442899712:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202709.889596:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442899840:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202709.889611:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202709.889622:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202709/real 1788202709] req@ffff902352689500 x1875064442899840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202709.889632:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442899840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202709.889639:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202709.889642:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202709.889645:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442899840:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202710.721003:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202711.745085:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202712.769035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202713.793013:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202714.817037:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202715.008937:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202715.008955:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202715.008960:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202715.008972:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1986 00000100:00080000:3.0:1788202715.008985:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202715.008993:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202715.009038:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442899968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202715.009068:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202715.009072:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202715.009074:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202715.009079:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1986 00000100:00080000:3.0:1788202715.009082:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202715.009087:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202715.009098:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202715.009112:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442900096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202715.009123:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202715.009126:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202715.009128:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202715.009134:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1986 00000100:00080000:3.0:1788202715.009138:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202715.009142:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202715.009149:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202715.009152:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442899968:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202715.009165:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442900224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202715.009179:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202715.009184:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202715.009186:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202715.009191:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1986 00000100:00080000:3.0:1788202715.009209:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202715.009219:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:2.0:1788202715.009221:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:3.0:1788202715.009235:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202715.009275:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442900352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202715.009292:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202715/real 1788202715] req@ffff902352689180 x1875064442899968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202715.009310:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442899968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202715.009318:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202715.009321:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202715.009324:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442899968:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202715.009329:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442900096:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202715.009345:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202715.009353:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202715/real 1788202715] req@ffff902352688700 x1875064442900096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202715.009360:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442900096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202715.009365:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202715.009367:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202715.009369:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442900096:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202715.009386:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442900224:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202715.009397:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202715.009402:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202715/real 1788202715] req@ffff90235268bb80 x1875064442900224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202715.009409:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442900224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202715.009418:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202715.009420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202715.009422:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442900224:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202715.009425:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442900352:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202715.009434:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202715.009438:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202715/real 1788202715] req@ffff902352689880 x1875064442900352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202715.009445:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442900352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202715.009449:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202715.009451:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202715.009452:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442900352:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202715.841013:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202716.864997:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202717.889050:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202718.913068:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202719.937058:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202720.128953:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202720.128962:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202720.128965:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202720.128973:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1991 00000100:00080000:3.0:1788202720.128983:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202720.128988:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202720.129020:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442900480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202720.129044:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202720.129046:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202720.129052:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202720.129055:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1991 00000100:00080000:3.0:1788202720.129064:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202720.129067:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202720.129077:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202720.129087:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442900608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202720.129092:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442900480:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202720.129105:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202720.129108:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202720.129109:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202720.129113:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1991 00000100:00080000:3.0:1788202720.129116:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202720.129118:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202720.129121:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202720.129134:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442900736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202720.129145:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202720.129149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202720.129151:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788202720.129169:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788202720.129184:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1991 00000100:00080000:3.0:1788202720.129191:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202720.129196:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202720.129201:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202720.129207:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202720/real 1788202720] req@ffff902352688380 x1875064442900480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202720.129215:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442900864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202720.129231:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442900480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202720.129241:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202720.129245:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202720.129248:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442900480:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202720.129266:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442900608:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202720.129283:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202720.129291:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202720/real 1788202720] req@ffff90235268ad80 x1875064442900608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202720.129300:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442900608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202720.129314:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202720.129316:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202720.129319:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442900608:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202720.129324:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442900736:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202720.129337:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202720.129344:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202720/real 1788202720] req@ffff90235268b480 x1875064442900736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202720.129352:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442900736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202720.129358:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202720.129360:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202720.129362:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442900736:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202720.129366:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442900864:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202720.129376:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202720.129382:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202720/real 1788202720] req@ffff902352688a80 x1875064442900864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202720.129390:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442900864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202720.129395:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202720.129397:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202720.129399:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442900864:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202720.961019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202721.985041:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202723.009002:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202724.033038:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202725.057054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202725.249017:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202725.249025:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202725.249029:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202725.249038:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 1996 00000100:00080000:2.0:1788202725.249051:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202725.249058:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202725.249116:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064442900992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202725.249142:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202725.249145:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202725.249147:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202725.249150:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1996 00000100:00080000:2.0:1788202725.249153:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202725.249156:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202725.249164:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202725.249173:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064442901120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202725.249182:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202725.249186:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788202725.249187:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442900992:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202725.249188:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202725.249196:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1996 00000100:00080000:2.0:1788202725.249200:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202725.249203:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202725.249206:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202725.249214:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442901248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202725.249221:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202725.249224:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202725.249225:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202725.249227:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 1996 00000100:00080000:2.0:1788202725.249229:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202725.249231:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202725.249234:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202725.249240:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442901376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202725.249268:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202725.249308:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202725/real 1788202725] req@ffff90234598ca80 x1875064442900992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202725.249327:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064442900992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202725.249338:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202725.249342:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202725.249346:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442900992:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202725.249370:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442901120:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202725.249391:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202725.249404:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202725/real 1788202725] req@ffff90234598e680 x1875064442901120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202725.249417:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064442901120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202725.249426:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202725.249429:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202725.249433:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442901120:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202725.249439:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442901248:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202725.249457:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202725.249467:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202725/real 1788202725] req@ffff90234598d880 x1875064442901248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202725.249478:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442901248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202725.249487:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202725.249490:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202725.249494:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442901248:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202725.249499:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442901376:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202725.249515:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202725.249525:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202725/real 1788202725] req@ffff90234598c700 x1875064442901376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202725.249536:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442901376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202725.249544:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202725.249547:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202725.249551:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442901376:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202726.081087:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202727.105102:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202728.129054:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202729.153136:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202730.177041:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202730.368959:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202730.368970:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202730.368974:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202730.369000:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2001 00000100:00080000:2.0:1788202730.369011:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202730.369018:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202730.369053:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442901504/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202730.369080:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202730.369084:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202730.369087:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202730.369092:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2001 00000100:00080000:2.0:1788202730.369096:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202730.369102:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202730.369112:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202730.369125:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442901504:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202730.369131:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442901632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202730.369146:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202730.369151:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202730.369153:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202730.369158:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2001 00000100:00080000:2.0:1788202730.369181:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788202730.369187:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788202730.369191:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202730.369198:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202730.369215:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442901760/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202730.369233:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202730.369237:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202730.369239:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202730.369243:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2001 00000100:00080000:2.0:1788202730.369248:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202730.369253:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202730.369258:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202730.369269:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442901888/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202730.369289:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202730/real 1788202730] req@ffff90234598ce00 x1875064442901504/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202730.369306:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442901504/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202730.369314:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202730.369317:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202730.369320:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442901504:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202730.369338:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442901632:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202730.369356:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202730.369365:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202730/real 1788202730] req@ffff90234598f800 x1875064442901632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202730.369488:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442901632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202730.369496:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202730.369499:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202730.369501:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442901632:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202730.369506:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442901760:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202730.369523:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202730.369580:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202730/real 1788202730] req@ffff90234598dc00 x1875064442901760/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202730.369595:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442901760/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202730.369602:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202730.369604:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202730.369607:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442901760:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202730.369611:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442901888:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202730.369625:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202730.369633:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202730/real 1788202730] req@ffff90234598f480 x1875064442901888/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202730.369640:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442901888/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202730.369645:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202730.369647:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202730.369648:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442901888:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202731.201046:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202732.225028:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202733.249042:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202734.273015:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202735.297057:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202735.488987:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202735.488998:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202735.489002:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202735.489012:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2007 00000100:00080000:0.0:1788202735.489023:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202735.489030:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202735.489068:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442902016/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202735.489105:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202735.489109:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202735.489111:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202735.489116:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2007 00000100:00080000:0.0:1788202735.489120:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202735.489125:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202735.489137:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202735.489145:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442902016:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202735.489153:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442902144/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202735.489167:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202735.489171:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202735.489174:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202735.489178:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2007 00000100:00080000:0.0:1788202735.489182:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202735.489186:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202735.489191:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202735.489202:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202735.489204:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442902272/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202735.489218:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202735.489222:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202735.489224:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202735.489229:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2007 00000100:00080000:0.0:1788202735.489233:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788202735.489238:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202735/real 1788202735] req@ffff9023450b8380 x1875064442902016/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202735.489240:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202735.489249:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202735.489258:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442902016/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202735.489264:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442902400/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202735.489269:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202735.489277:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202735.489283:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442902016:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202735.489304:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442902144:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202735.489318:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202735.489326:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202735/real 1788202735] req@ffff9023450b8a80 x1875064442902144/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202735.489335:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442902144/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202735.489340:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202735.489342:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202735.489344:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442902144:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202735.489348:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442902272:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202735.489358:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202735.489363:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202735/real 1788202735] req@ffff9023450bb800 x1875064442902272/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202735.489370:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442902272/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202735.489376:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202735.489377:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202735.489379:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442902272:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202735.489383:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442902400:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202735.489392:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202735.489397:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202735/real 1788202735] req@ffff9023450b9500 x1875064442902400/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202735.489404:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442902400/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202735.489409:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202735.489411:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202735.489413:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442902400:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202736.321018:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202737.345040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202738.369028:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202739.393018:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202740.417003:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202740.608989:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202740.608997:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202740.609000:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202740.609009:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2012 00000100:00080000:0.0:1788202740.609019:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202740.609025:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202740.609063:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442902528/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202740.609086:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202740.609089:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202740.609091:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202740.609095:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2012 00000100:00080000:0.0:1788202740.609099:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202740.609103:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202740.609121:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202740.609131:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442902528:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202740.609139:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442902656/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202740.609152:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202740.609155:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202740.609156:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202740.609159:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2012 00000100:00080000:0.0:1788202740.609162:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202740.609166:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202740.609170:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202740.609180:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442902784/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202740.609191:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202740.609193:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202740.609195:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202740.609200:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2012 00000800:00000100:3.0:1788202740.609201:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788202740.609203:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202740.609208:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202740.609212:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202740.609224:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442902912/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202740.609242:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202740/real 1788202740] req@ffff9023450b8700 x1875064442902528/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202740.609264:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442902528/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202740.609276:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202740.609280:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202740.609285:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442902528:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202740.609308:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442902656:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202740.609327:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202740.609336:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202740/real 1788202740] req@ffff9023450b9c00 x1875064442902656/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202740.609359:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442902656/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202740.609368:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202740.609372:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202740.609375:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442902656:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202740.609381:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442902784:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202740.609404:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202740.609417:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202740/real 1788202740] req@ffff9023450b9f80 x1875064442902784/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202740.609426:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442902784/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202740.609432:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202740.609434:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202740.609436:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442902784:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202740.609441:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442902912:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202740.609454:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202740.609461:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202740/real 1788202740] req@ffff9023450bad80 x1875064442902912/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202740.609468:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442902912/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202740.609478:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202740.609480:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202740.609482:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442902912:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202741.441045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202742.465021:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202743.489064:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202744.513053:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202745.537045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202745.728973:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202745.728981:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202745.728984:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202745.728992:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2017 00000100:00080000:1.0:1788202745.729003:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202745.729008:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202745.729042:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442903040/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202745.729066:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202745.729069:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202745.729070:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202745.729073:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2017 00000100:00080000:1.0:1788202745.729075:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202745.729077:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202745.729084:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202745.729093:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442903168/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202745.729099:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202745.729102:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202745.729103:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202745.729106:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2017 00000100:00080000:1.0:1788202745.729108:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202745.729111:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202745.729116:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202745.729127:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442903296/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202745.729134:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442903040:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202745.729140:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202745.729146:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202745.729148:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202745.729163:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2017 00000100:00080000:1.0:1788202745.729168:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202745.729173:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202745.729178:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202745.729191:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442903424/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202745.729207:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202745.729264:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202745/real 1788202745] req@ffff90234519ed80 x1875064442903040/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202745.729286:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442903040/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202745.729297:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202745.729301:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202745.729306:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442903040:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202745.729314:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442903168:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202745.729338:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202745.729352:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202745/real 1788202745] req@ffff90234519d880 x1875064442903168/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202745.729367:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442903168/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202745.729376:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202745.729380:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202745.729384:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442903168:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202745.729409:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442903296:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202745.729432:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202745.729445:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202745/real 1788202745] req@ffff90234519c000 x1875064442903296/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202745.729457:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442903296/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202745.729466:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202745.729469:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202745.729473:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442903296:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202745.729480:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442903424:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202745.729501:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202745.729514:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202745/real 1788202745] req@ffff90234519e300 x1875064442903424/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202745.729527:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442903424/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202745.729535:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202745.729539:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202745.729543:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442903424:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202746.561049:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202747.585046:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202748.609049:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202749.633017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202750.657066:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202750.849009:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202750.849021:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202750.849025:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202750.849036:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2022 00000100:00080000:3.0:1788202750.849048:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202750.849055:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202750.849095:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064442903552/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202750.849126:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202750.849129:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202750.849131:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202750.849134:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2022 00000100:00080000:3.0:1788202750.849137:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202750.849140:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202750.849148:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202750.849161:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442903680/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202750.849169:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202750.849172:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202750.849175:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202750.849177:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442903552:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202750.849180:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2022 00000100:00080000:3.0:1788202750.849185:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202750.849189:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202750.849193:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202750.849200:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442903808/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202750.849211:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202750.849214:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202750.849216:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202750.849218:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2022 00000100:00080000:3.0:1788202750.849234:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202750.849236:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202750.849239:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788202750.849245:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202750.849249:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442903936/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202750.849290:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202750/real 1788202750] req@ffff90235268aa00 x1875064442903552/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202750.849310:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064442903552/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202750.849321:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202750.849325:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202750.849329:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442903552:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202750.849351:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442903680:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202750.849371:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202750.849382:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202750/real 1788202750] req@ffff902352688a80 x1875064442903680/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202750.849395:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442903680/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202750.849404:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202750.849408:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202750.849411:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442903680:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202750.849417:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442903808:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202750.849433:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202750.849442:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202750/real 1788202750] req@ffff90235268b480 x1875064442903808/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202750.849454:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442903808/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202750.849462:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202750.849466:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202750.849469:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442903808:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202750.849475:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442903936:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202750.849490:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202750.849500:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202750/real 1788202750] req@ffff90235268ad80 x1875064442903936/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202750.849511:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442903936/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202750.849525:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202750.849529:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202750.849532:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442903936:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202751.681029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202752.705004:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202753.729038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202754.753041:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202755.777099:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202755.968978:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202755.968993:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202755.968998:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202755.969009:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2027 00000100:00080000:0.0:1788202755.969022:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202755.969029:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202755.969074:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442904064/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202755.969103:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202755.969106:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202755.969107:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202755.969111:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2027 00000100:00080000:0.0:1788202755.969113:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202755.969115:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202755.969127:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202755.969142:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442904192/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202755.969157:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202755.969168:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788202755.969169:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442904064:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202755.969171:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202755.969185:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2027 00000100:00080000:0.0:1788202755.969191:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202755.969197:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202755.969202:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202755.969215:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442904320/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202755.969223:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202755.969227:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202755.969228:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202755.969232:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2027 00000100:00080000:0.0:1788202755.969235:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202755.969238:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202755.969243:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202755.969252:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442904448/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202755.969262:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202755.969315:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202755/real 1788202755] req@ffff9023450baa00 x1875064442904064/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202755.969336:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442904064/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202755.969359:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202755.969367:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202755.969372:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442904064:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202755.969399:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442904192:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202755.969423:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202755.969436:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202755/real 1788202755] req@ffff9023450b9180 x1875064442904192/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202755.969450:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442904192/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202755.969460:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202755.969464:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202755.969468:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442904192:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202755.969476:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442904320:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202755.969495:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202755.969506:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202755/real 1788202755] req@ffff9023450b8000 x1875064442904320/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202755.969519:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442904320/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202755.969528:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202755.969533:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202755.969566:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442904320:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202755.969574:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442904448:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202755.969595:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202755.969606:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202755/real 1788202755] req@ffff9023450ba300 x1875064442904448/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202755.969619:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442904448/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202755.969628:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202755.969632:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202755.969635:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442904448:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202756.801102:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202757.825051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202758.849079:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202759.872990:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202760.897039:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202761.088977:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202761.088985:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202761.088989:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202761.088998:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2032 00000100:00080000:2.0:1788202761.089008:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202761.089013:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202761.089047:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442904576/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202761.089079:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202761.089082:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202761.089083:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202761.089087:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2032 00000100:00080000:2.0:1788202761.089089:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202761.089091:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202761.089099:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202761.089109:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442904704/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202761.089117:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202761.089119:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202761.089120:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202761.089123:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2032 00000100:00080000:2.0:1788202761.089124:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202761.089127:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202761.089129:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202761.089136:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442904832/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202761.089149:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442904576:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202761.089150:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202761.089157:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202761.089161:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202761.089164:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2032 00000100:00080000:2.0:1788202761.089166:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202761.089168:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202761.089171:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202761.089187:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442904960/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202761.089230:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202761.089276:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202761/real 1788202761] req@ffff90234598c380 x1875064442904576/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202761.089297:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442904576/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202761.089309:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202761.089312:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202761.089315:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442904576:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202761.089321:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442904704:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202761.089340:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202761.089349:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202761/real 1788202761] req@ffff90234598d180 x1875064442904704/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202761.089358:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442904704/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202761.089363:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202761.089365:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202761.089367:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442904704:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202761.089387:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442904832:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202761.089397:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202761.089404:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202761/real 1788202761] req@ffff90234598e300 x1875064442904832/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202761.089411:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442904832/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202761.089416:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202761.089418:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202761.089420:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442904832:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202761.089423:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442904960:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202761.089432:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202761.089438:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202761/real 1788202761] req@ffff90234598df80 x1875064442904960/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202761.089467:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442904960/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202761.089479:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202761.089483:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202761.089486:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442904960:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202761.920995:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202762.945033:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202763.969050:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202764.993066:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202766.017125:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202766.209035:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202766.209046:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202766.209051:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202766.209063:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2037 00000100:00080000:0.0:1788202766.209076:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202766.209084:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202766.209135:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442905088/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202766.209169:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202766.209172:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202766.209173:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202766.209177:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2037 00000100:00080000:0.0:1788202766.209180:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202766.209184:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202766.209194:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202766.209210:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442905216/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202766.209217:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202766.209220:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202766.209221:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202766.209224:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2037 00000100:00080000:0.0:1788202766.209226:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202766.209228:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202766.209232:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202766.209239:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442905088:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202766.209243:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442905344/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202766.209259:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202766.209261:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202766.209262:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202766.209265:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2037 00000100:00080000:0.0:1788202766.209267:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202766.209269:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202766.209272:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202766.209280:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442905472/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202766.209334:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202766.209420:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202766/real 1788202766] req@ffff9023450ba300 x1875064442905088/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202766.209444:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442905088/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202766.209455:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202766.209458:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202766.209461:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442905088:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202766.209468:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442905216:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202766.209490:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202766.209502:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202766/real 1788202766] req@ffff9023450b8000 x1875064442905216/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202766.209514:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442905216/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202766.209520:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202766.209523:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202766.209525:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442905216:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202766.209561:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442905344:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202766.209578:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202766.209602:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202766/real 1788202766] req@ffff9023450b9180 x1875064442905344/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202766.209617:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442905344/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202766.209631:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202766.209637:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202766.209641:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442905344:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202766.209718:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442905472:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202766.209749:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202766.209766:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202766/real 1788202766] req@ffff9023450baa00 x1875064442905472/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202766.209784:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442905472/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202766.209794:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202766.209798:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202766.209802:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442905472:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202767.041199:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202768.065090:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202769.089059:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202770.113080:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202771.137094:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202771.329009:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202771.329020:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202771.329025:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202771.329034:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2042 00000100:00080000:1.0:1788202771.329045:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202771.329052:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202771.329088:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442905600/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202771.329114:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202771.329121:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202771.329123:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202771.329127:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2042 00000100:00080000:1.0:1788202771.329132:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202771.329137:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202771.329147:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202771.329160:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442905600:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202771.329171:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442905728/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202771.329185:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202771.329189:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202771.329192:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202771.329197:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2042 00000100:00080000:1.0:1788202771.329200:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202771.329203:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202771.329208:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202771.329217:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442905856/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202771.329224:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202771.329227:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202771.329228:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202771.329231:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2042 00000100:00080000:1.0:1788202771.329233:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202771.329236:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202771.329240:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202771.329249:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442905984/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202771.329293:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442905728:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202771.329322:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442905856:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202771.329336:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442905984:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202772.161112:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202773.185000:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202774.209012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788202774.401179:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788202774.401240:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788202774.401257:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202774.401299:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202774.401309:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202774.401317:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788202774.401358:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202771/real 1788202774] req@ffff90234519e300 x1875064442905728/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202774.401387:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442905728/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202774.401400:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202774.401405:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202774.401410:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442905728:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202774.401418:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202771/real 1788202774] req@ffff90234519c000 x1875064442905856/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202774.401431:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442905856/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202774.401440:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202774.401444:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202774.401447:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442905856:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202774.401453:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202771/real 1788202774] req@ffff90234519d880 x1875064442905984/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202774.401466:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442905984/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202774.401480:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202774.401484:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202774.401488:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442905984:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202774.401495:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202771/real 1788202774] req@ffff90234519f800 x1875064442905600/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202774.401508:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442905600/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202774.401517:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202774.401521:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202774.401531:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442905600:192.168.201.130@tcp:250:kworker.0 00010000:00010000:2.0:1788202775.233003:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202776.257005:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202776.449007:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202776.449017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202776.449020:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202776.449029:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2047 00000100:00080000:3.0:1788202776.449038:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202776.449044:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202776.449072:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442906112/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202776.449097:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202776.449101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202776.449103:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202776.449107:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2047 00000100:00080000:3.0:1788202776.449112:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202776.449116:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202776.449126:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202776.449135:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442906112:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202776.449144:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442906240/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202776.449165:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202776.449169:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202776.449171:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202776.449175:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2047 00000100:00080000:3.0:1788202776.449192:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202776.449198:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202776.449204:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202776.449217:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442906368/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202776.449228:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788202776.449230:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788202776.449236:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202776.449239:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202776.449244:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2047 00000100:00080000:3.0:1788202776.449257:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202776.449262:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202776.449268:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202776.449274:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202776/real 1788202776] req@ffff902352688380 x1875064442906112/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202776.449284:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442906496/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202776.449300:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442906112/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202776.449322:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202776.449327:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202776.449331:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442906112:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202776.449357:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442906240:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202776.449380:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202776.449393:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202776/real 1788202776] req@ffff902352689880 x1875064442906240/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202776.449407:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442906240/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202776.449416:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202776.449419:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202776.449422:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442906240:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202776.449429:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442906368:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202776.449457:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202776.449469:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202776/real 1788202776] req@ffff90235268bb80 x1875064442906368/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202776.449489:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442906368/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202776.449499:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202776.449502:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202776.449506:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442906368:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202776.449513:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442906496:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202776.449540:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202776.449552:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202776/real 1788202776] req@ffff902352688700 x1875064442906496/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202776.449563:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442906496/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202776.449572:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202776.449576:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202776.449579:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442906496:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202777.281013:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202778.305047:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202779.329034:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202780.353092:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202781.377091:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202781.568994:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202781.569003:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202781.569007:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202781.569015:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2053 00000100:00080000:3.0:1788202781.569024:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202781.569029:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202781.569061:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442906624/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202781.569086:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202781.569101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202781.569102:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202781.569106:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2053 00000100:00080000:3.0:1788202781.569109:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202781.569112:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202781.569120:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202781.569131:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442906752/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202781.569140:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202781.569144:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788202781.569146:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442906624:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202781.569146:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202781.569156:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2053 00000100:00080000:3.0:1788202781.569159:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202781.569161:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202781.569166:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202781.569176:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442906880/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202781.569184:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202781.569186:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202781.569188:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202781.569190:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2053 00000100:00080000:3.0:1788202781.569200:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202781.569203:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202781.569206:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788202781.569207:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202781.569218:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442907008/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202781.569244:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202781/real 1788202781] req@ffff902352689180 x1875064442906624/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202781.569260:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442906624/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202781.569269:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202781.569272:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202781.569275:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442906624:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202781.569293:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442906752:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202781.569308:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202781.569317:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202781/real 1788202781] req@ffff902352689500 x1875064442906752/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202781.569325:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442906752/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202781.569332:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202781.569333:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202781.569336:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442906752:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202781.569340:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442906880:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202781.569351:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202781.569357:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202781/real 1788202781] req@ffff90235268a680 x1875064442906880/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202781.569365:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442906880/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202781.569371:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202781.569372:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202781.569374:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442906880:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202781.569377:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442907008:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202781.569386:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202781.569392:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202781/real 1788202781] req@ffff902352689c00 x1875064442907008/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202781.569399:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442907008/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202781.569405:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202781.569407:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202781.569409:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442907008:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202782.401071:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202783.425021:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202784.449059:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202785.473033:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202786.497042:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202786.688990:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202786.688998:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202786.689025:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202786.689035:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2058 00000100:00080000:1.0:1788202786.689046:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202786.689053:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202786.689094:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442907136/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202786.689132:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202786.689135:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202786.689137:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202786.689140:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2058 00000100:00080000:1.0:1788202786.689143:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202786.689147:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202786.689156:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202786.689167:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442907264/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202786.689177:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202786.689179:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202786.689181:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202786.689184:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2058 00000100:00080000:1.0:1788202786.689188:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202786.689191:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202786.689196:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202786.689212:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442907392/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202786.689218:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442907136:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202786.689233:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202786.689239:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202786.689240:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202786.689245:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2058 00000100:00080000:1.0:1788202786.689248:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202786.689251:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202786.689257:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202786.689267:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442907520/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202786.689326:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202786.689379:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202786/real 1788202786] req@ffff90234519ed80 x1875064442907136/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202786.689395:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442907136/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202786.689405:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202786.689409:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202786.689411:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442907136:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202786.689418:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442907264:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202786.689437:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202786.689446:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202786/real 1788202786] req@ffff90234519df80 x1875064442907264/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202786.689454:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442907264/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202786.689460:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202786.689462:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202786.689464:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442907264:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202786.689489:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442907392:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202786.689502:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202786.689509:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202786/real 1788202786] req@ffff90234519fb80 x1875064442907392/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202786.689517:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442907392/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202786.689522:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202786.689524:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202786.689526:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442907392:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202786.689530:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442907520:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202786.689540:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202786.689546:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202786/real 1788202786] req@ffff90234519c700 x1875064442907520/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202786.689553:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442907520/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202786.689558:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202786.689560:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202786.689561:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442907520:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202787.521088:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202788.545062:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202789.569067:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202790.593083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202791.617111:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202791.809033:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202791.809044:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202791.809049:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202791.809060:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2063 00000100:00080000:0.0:1788202791.809072:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202791.809080:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202791.809127:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442907648/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202791.809160:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202791.809164:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202791.809166:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202791.809171:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2063 00000100:00080000:0.0:1788202791.809176:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202791.809180:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202791.809191:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202791.809204:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442907648:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202791.809205:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442907776/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202791.809219:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202791.809222:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202791.809223:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202791.809226:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2063 00000100:00080000:0.0:1788202791.809228:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202791.809231:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202791.809234:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202791.809242:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442907904/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202791.809249:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202791.809252:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202791.809253:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202791.809256:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2063 00000100:00080000:0.0:1788202791.809259:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202791.809262:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202791.809264:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202791.809271:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442908032/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202791.809292:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202791.809338:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202791/real 1788202791] req@ffff9023450bad80 x1875064442907648/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202791.809363:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442907648/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202791.809372:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202791.809375:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202791.809378:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442907648:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202791.809403:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442907776:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202791.809420:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202791.809428:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202791/real 1788202791] req@ffff9023450b9f80 x1875064442907776/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202791.809437:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442907776/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202791.809442:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202791.809445:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202791.809447:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442907776:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202791.809450:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442907904:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202791.809461:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202791.809467:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202791/real 1788202791] req@ffff9023450b9c00 x1875064442907904/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202791.809479:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442907904/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202791.809484:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202791.809486:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202791.809488:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442907904:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202791.809491:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442908032:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202791.809502:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202791.809507:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202791/real 1788202791] req@ffff9023450b8700 x1875064442908032/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202791.809514:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442908032/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202791.809519:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202791.809521:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202791.809523:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442908032:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202792.641063:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202793.665046:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202794.689080:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202795.713102:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202796.737035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202796.929076:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202796.929088:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202796.929092:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202796.929103:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2068 00000100:00080000:2.0:1788202796.929117:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202796.929124:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202796.929188:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442908160/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202796.929219:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202796.929222:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202796.929223:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202796.929227:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2068 00000100:00080000:2.0:1788202796.929230:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202796.929233:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202796.929243:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202796.929255:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442908288/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202796.929262:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202796.929265:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202796.929266:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202796.929269:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2068 00000100:00080000:2.0:1788202796.929271:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202796.929273:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202796.929277:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202796.929286:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442908416/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202796.929300:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788202796.929300:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442908160:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202796.929313:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202796.929315:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202796.929319:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2068 00000100:00080000:2.0:1788202796.929322:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202796.929325:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202796.929329:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202796.929340:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442908544/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202796.929393:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202796.929448:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202796/real 1788202796] req@ffff90234598fb80 x1875064442908160/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202796.929470:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442908160/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202796.929479:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202796.929483:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202796.929486:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442908160:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202796.929492:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442908288:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202796.929513:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202796.929522:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202796/real 1788202796] req@ffff90234598ed80 x1875064442908288/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202796.929530:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442908288/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202796.929536:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202796.929539:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202796.929541:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442908288:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202796.929564:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442908416:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202796.929584:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202796.929596:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202796/real 1788202796] req@ffff90234598f100 x1875064442908416/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202796.929610:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442908416/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202796.929617:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202796.929620:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202796.929622:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442908416:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202796.929627:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442908544:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202796.929642:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202796.929649:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202796/real 1788202796] req@ffff90234598df80 x1875064442908544/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202796.929662:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442908544/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202796.929667:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202796.929669:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202796.929671:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442908544:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202797.761049:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202798.785021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202799.809099:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202800.833033:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202801.857065:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202802.048957:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202802.048969:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202802.048974:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202802.048985:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2073 00000100:00080000:2.0:1788202802.048995:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202802.049003:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202802.049041:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442908672/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202802.049075:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202802.049080:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202802.049082:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202802.049089:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2073 00000100:00080000:2.0:1788202802.049092:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202802.049099:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202802.049107:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202802.049121:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442908800/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202802.049134:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202802.049137:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202802.049139:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202802.049142:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2073 00000100:00080000:2.0:1788202802.049145:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202802.049148:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202802.049151:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202802.049164:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442908928/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202802.049165:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442908672:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202802.049180:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202802.049183:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202802.049185:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202802.049188:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2073 00000100:00080000:2.0:1788202802.049196:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202802.049199:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202802.049203:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202802.049217:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442909056/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202802.049233:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202802.049361:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202802/real 1788202802] req@ffff90234598e300 x1875064442908672/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202802.049376:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442908672/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202802.049385:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202802.049388:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202802.049391:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442908672:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202802.049397:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442908800:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202802.049416:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202802.049428:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202802/real 1788202802] req@ffff90234598d180 x1875064442908800/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202802.049437:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442908800/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202802.049447:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202802.049450:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202802.049452:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442908800:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202802.049474:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442908928:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202802.049488:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202802.049604:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202802/real 1788202802] req@ffff90234598c380 x1875064442908928/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202802.049615:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442908928/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202802.049622:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202802.049625:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202802.049627:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442908928:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202802.049664:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442909056:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202802.049684:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202802.049697:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202802/real 1788202802] req@ffff90234598f480 x1875064442909056/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202802.049705:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442909056/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202802.049713:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202802.049716:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202802.049721:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442909056:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202802.881049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202803.905069:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202804.929085:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202805.953053:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202806.977105:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202807.169049:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202807.169066:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202807.169071:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202807.169082:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2078 00000100:00080000:0.0:1788202807.169095:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202807.169103:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202807.169147:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442909184/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202807.169175:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202807.169178:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202807.169179:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202807.169182:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2078 00000100:00080000:0.0:1788202807.169185:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202807.169187:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202807.169197:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202807.169207:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442909312/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202807.169221:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202807.169224:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202807.169225:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202807.169228:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2078 00000100:00080000:0.0:1788202807.169231:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202807.169233:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202807.169237:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202807.169242:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442909184:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202807.169251:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442909440/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202807.169260:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202807.169263:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202807.169264:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202807.169267:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2078 00000100:00080000:0.0:1788202807.169270:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202807.169272:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202807.169275:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202807.169284:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442909568/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202807.169340:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202807.169396:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202807/real 1788202807] req@ffff9023450b9500 x1875064442909184/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202807.169424:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442909184/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202807.169437:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202807.169441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202807.169446:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442909184:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202807.169454:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442909312:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202807.169477:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202807.169489:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202807/real 1788202807] req@ffff9023450bb800 x1875064442909312/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202807.169502:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442909312/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202807.169511:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202807.169514:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202807.169518:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442909312:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202807.169547:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442909440:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202807.169565:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202807.169576:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202807/real 1788202807] req@ffff9023450b8a80 x1875064442909440/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202807.169588:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442909440/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202807.169597:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202807.169600:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202807.169604:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442909440:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202807.169609:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442909568:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202807.169634:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202807.169655:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202807/real 1788202807] req@ffff9023450b8380 x1875064442909568/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202807.169669:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442909568/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202807.169677:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202807.169680:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202807.169684:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442909568:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202808.001031:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202809.025081:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202810.049044:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202811.073064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202812.097083:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202812.288972:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202812.288982:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202812.288987:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202812.288997:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2083 00000100:00080000:1.0:1788202812.289009:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202812.289016:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202812.289053:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442909696/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202812.289080:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202812.289084:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202812.289086:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202812.289091:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2083 00000100:00080000:1.0:1788202812.289094:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202812.289099:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202812.289108:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202812.289121:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442909824/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202812.289127:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442909696:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202812.289138:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202812.289142:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202812.289144:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202812.289148:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2083 00000100:00080000:1.0:1788202812.289152:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202812.289155:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202812.289160:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202812.289172:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442909952/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202812.289178:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202812.289180:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202812.289181:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202812.289183:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2083 00000100:00080000:1.0:1788202812.289186:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202812.289189:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202812.289192:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202812.289200:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202812.289204:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442910080/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202812.289256:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202812/real 1788202812] req@ffff90234519ea00 x1875064442909696/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202812.289282:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442909696/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202812.289295:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202812.289301:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202812.289305:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442909696:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202812.289331:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442909824:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202812.289355:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202812.289368:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202812/real 1788202812] req@ffff90234519c380 x1875064442909824/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202812.289380:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442909824/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202812.289389:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202812.289392:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202812.289395:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442909824:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202812.289401:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442909952:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202812.289421:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202812.289432:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202812/real 1788202812] req@ffff90234519f480 x1875064442909952/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202812.289444:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442909952/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202812.289453:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202812.289456:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202812.289460:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442909952:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202812.289479:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442910080:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202812.289501:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202812.289514:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202812/real 1788202812] req@ffff90234519dc00 x1875064442910080/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202812.289527:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442910080/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202812.289536:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202812.289540:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202812.289543:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442910080:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202813.121057:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202814.145035:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202815.168994:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202816.193015:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202817.217055:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202817.408971:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202817.408980:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202817.408984:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202817.408992:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2088 00000100:00080000:2.0:1788202817.409002:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202817.409008:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202817.409040:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442910208/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202817.409063:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202817.409067:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202817.409069:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202817.409074:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2088 00000100:00080000:2.0:1788202817.409077:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202817.409082:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202817.409092:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202817.409109:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442910336/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202817.409118:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442910208:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202817.409124:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202817.409128:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202817.409129:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202817.409133:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2088 00000100:00080000:2.0:1788202817.409136:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202817.409140:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202817.409144:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202817.409156:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442910464/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202817.409163:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202817.409165:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202817.409167:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202817.409170:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2088 00000100:00080000:2.0:1788202817.409172:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202817.409174:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202817.409177:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202817.409181:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202817.409189:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442910592/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202817.409218:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202817/real 1788202817] req@ffff90234598dc00 x1875064442910208/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202817.409232:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442910208/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202817.409240:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202817.409243:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202817.409246:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442910208:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202817.409264:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442910336:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202817.409279:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202817.409287:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202817/real 1788202817] req@ffff90234598f800 x1875064442910336/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202817.409295:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442910336/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202817.409301:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202817.409303:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202817.409305:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442910336:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202817.409308:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442910464:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202817.409319:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202817.409325:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202817/real 1788202817] req@ffff90234598ce00 x1875064442910464/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202817.409333:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442910464/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202817.409338:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202817.409340:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202817.409342:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442910464:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202817.409345:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442910592:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202817.409355:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202817.409360:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202817/real 1788202817] req@ffff90234598c700 x1875064442910592/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202817.409367:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442910592/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202817.409373:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202817.409375:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202817.409377:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442910592:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202818.241005:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202819.265002:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202820.289107:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202821.313035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202822.337037:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202822.528988:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202822.528997:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202822.529001:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202822.529010:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2094 00000100:00080000:2.0:1788202822.529020:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202822.529026:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202822.529059:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442910720/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202822.529085:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202822.529088:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202822.529089:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202822.529094:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2094 00000100:00080000:2.0:1788202822.529096:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202822.529099:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202822.529107:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202822.529116:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064442910848/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202822.529123:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202822.529126:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202822.529127:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202822.529129:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2094 00000100:00080000:2.0:1788202822.529130:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202822.529133:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202822.529136:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202822.529146:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064442910976/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202822.529156:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202822.529160:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202822.529162:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202822.529166:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2094 00000100:00080000:2.0:1788202822.529171:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788202822.529172:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442910720:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202822.529180:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202822.529186:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202822.529199:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442911104/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202822.529233:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202822.529266:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202822/real 1788202822] req@ffff90234598d880 x1875064442910720/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202822.529280:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442910720/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202822.529288:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202822.529291:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202822.529294:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442910720:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202822.529298:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442910848:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202822.529311:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202822.529318:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202822/real 1788202822] req@ffff90234598e680 x1875064442910848/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202822.529326:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064442910848/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202822.529331:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202822.529333:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202822.529335:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442910848:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202822.529339:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442910976:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202822.529348:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202822.529353:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202822/real 1788202822] req@ffff90234598ca80 x1875064442910976/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202822.529360:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064442910976/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202822.529365:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202822.529367:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202822.529369:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442910976:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202822.529416:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442911104:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202822.529430:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202822.529437:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202822/real 1788202822] req@ffff90234598c700 x1875064442911104/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202822.529445:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442911104/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202822.529450:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202822.529452:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202822.529454:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442911104:192.168.201.130@tcp:8:kworker.0 00010000:00010000:3.0:1788202823.360986:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202824.385045:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202825.409070:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202826.433043:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202827.456982:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202827.648989:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202827.648997:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202827.649000:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202827.649014:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2099 00000100:00080000:3.0:1788202827.649028:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202827.649033:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202827.649068:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442911232/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202827.649099:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202827.649103:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202827.649105:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202827.649110:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2099 00000100:00080000:3.0:1788202827.649115:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202827.649121:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202827.649131:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202827.649133:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442911232:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202827.649149:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442911360/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202827.649163:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202827.649168:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202827.649169:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202827.649173:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2099 00000100:00080000:3.0:1788202827.649185:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202827.649189:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:2.0:1788202827.649193:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:3.0:1788202827.649197:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202827.649212:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442911488/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202827.649220:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202827.649223:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202827.649225:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202827.649228:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2099 00000100:00100000:2.0:1788202827.649231:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202827/real 1788202827] req@ffff902352688e00 x1875064442911232/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202827.649233:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202827.649241:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202827.649250:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202827.649257:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442911232/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202827.649261:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442911616/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202827.649285:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202827.649308:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202827.649313:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442911232:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202827.649340:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442911360:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202827.649360:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202827.649369:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202827/real 1788202827] req@ffff90235268a300 x1875064442911360/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202827.649379:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442911360/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202827.649385:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202827.649387:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202827.649390:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442911360:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202827.649394:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442911488:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202827.649406:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202827.649412:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202827/real 1788202827] req@ffff902352688000 x1875064442911488/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202827.649420:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442911488/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202827.649426:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202827.649428:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202827.649430:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442911488:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202827.649433:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442911616:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202827.649443:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202827.649448:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202827/real 1788202827] req@ffff90235268b800 x1875064442911616/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202827.649456:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442911616/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202827.649461:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202827.649463:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202827.649464:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442911616:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202828.481104:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202829.505036:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202830.529104:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202831.553017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202832.577193:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202832.769017:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202832.769029:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202832.769033:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202832.769042:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2104 00000100:00080000:1.0:1788202832.769053:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202832.769059:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202832.769094:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442911744/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202832.769121:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202832.769124:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202832.769126:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202832.769129:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2104 00000100:00080000:1.0:1788202832.769131:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202832.769134:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202832.769142:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202832.769153:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442911872/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202832.769161:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202832.769164:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202832.769165:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202832.769168:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2104 00000100:00080000:1.0:1788202832.769171:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202832.769176:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202832.769181:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202832.769189:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442912000/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202832.769196:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202832.769199:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202832.769213:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202832.769218:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2104 00000100:00080000:1.0:1788202832.769222:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202832.769231:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202832.769231:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442911744:192.168.201.130@tcp:250:kworker.0 00000008:00100000:1.0:1788202832.769238:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202832.769257:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442912128/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202832.769332:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202832.769380:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202832/real 1788202832] req@ffff90234519f100 x1875064442911744/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202832.769396:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442911744/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202832.769405:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202832.769409:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202832.769412:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442911744:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202832.769419:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442911872:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202832.769445:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202832.769455:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202832/real 1788202832] req@ffff90234519dc00 x1875064442911872/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202832.769463:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442911872/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202832.769469:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202832.769472:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202832.769474:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442911872:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202832.769478:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442912000:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202832.769490:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202832.769497:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202832/real 1788202832] req@ffff90234519f480 x1875064442912000/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202832.769505:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442912000/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202832.769515:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202832.769517:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202832.769519:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442912000:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202832.769557:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442912128:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202832.769578:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202832.769590:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202832/real 1788202832] req@ffff90234519c380 x1875064442912128/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202832.769598:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442912128/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202832.769603:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202832.769605:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202832.769608:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442912128:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202833.601033:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202834.625092:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202835.648998:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202836.673078:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202837.697088:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202837.889034:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202837.889047:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202837.889051:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202837.889063:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2109 00000100:00080000:3.0:1788202837.889075:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202837.889083:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202837.889155:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442912256/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202837.889196:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202837.889201:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202837.889202:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202837.889208:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2109 00000100:00080000:3.0:1788202837.889213:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202837.889218:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202837.889229:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202837.889244:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442912256:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202837.889247:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442912384/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202837.889268:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202837.889272:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202837.889275:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202837.889280:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2109 00000100:00080000:3.0:1788202837.889284:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202837.889290:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202837.889297:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202837.889309:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442912512/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202837.889317:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202837.889321:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202837.889324:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202837.889328:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2109 00000100:00080000:3.0:1788202837.889333:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202837.889338:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202837.889345:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202837.889388:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442912384:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202837.889424:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442912512:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202837.889435:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442912640/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202837.889466:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442912640:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202838.721031:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202839.744981:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202840.769045:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788202840.961209:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:0.0:1788202840.961257:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788202840.961275:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788202840.961321:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788202840.961331:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788202840.961341:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:1.0:1788202840.961377:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202837/real 1788202840] req@ffff90235268a300 x1875064442912640/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202840.961409:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442912640/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202840.961421:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202840.961426:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202840.961431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442912640:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202840.961440:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202837/real 1788202840] req@ffff90235268b800 x1875064442912384/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202840.961455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442912384/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202840.961464:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202840.961468:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202840.961471:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442912384:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202840.961478:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202837/real 1788202840] req@ffff902352688000 x1875064442912512/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202840.961490:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442912512/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202840.961500:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202840.961504:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202840.961508:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442912512:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202840.961515:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202837/real 1788202840] req@ffff90235268b100 x1875064442912256/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202840.961530:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442912256/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202840.961539:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202840.961543:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202840.961547:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442912256:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788202841.793061:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202842.817062:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202843.009011:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202843.009020:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202843.009038:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202843.009049:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2114 00000100:00080000:2.0:1788202843.009061:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202843.009069:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202843.009110:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442912768/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202843.009136:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202843.009139:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202843.009140:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202843.009143:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2114 00000100:00080000:2.0:1788202843.009146:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202843.009148:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202843.009158:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202843.009167:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442912896/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202843.009177:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202843.009181:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202843.009184:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202843.009188:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2114 00000100:00080000:2.0:1788202843.009191:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202843.009197:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442912768:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788202843.009199:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202843.009209:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202843.009225:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442913024/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202843.009235:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202843.009239:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202843.009241:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202843.009245:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2114 00000100:00080000:2.0:1788202843.009249:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202843.009268:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788202843.009274:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202843.009287:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442913152/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202843.009305:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202843.009363:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202843/real 1788202843] req@ffff90234598ce00 x1875064442912768/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202843.009386:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442912768/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202843.009401:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202843.009406:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202843.009411:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442912768:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202843.009420:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442912896:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202843.009447:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202843.009461:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202843/real 1788202843] req@ffff90234598f800 x1875064442912896/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202843.009475:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442912896/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202843.009491:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202843.009495:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202843.009498:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442912896:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202843.009535:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442913024:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202843.009556:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202843.009566:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202843/real 1788202843] req@ffff90234598dc00 x1875064442913024/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202843.009578:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442913024/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202843.009588:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202843.009591:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202843.009595:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442913024:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202843.009601:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442913152:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202843.009620:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202843.009632:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202843/real 1788202843] req@ffff90234598f480 x1875064442913152/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202843.009644:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442913152/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202843.009653:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202843.009656:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202843.009661:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442913152:192.168.201.130@tcp:8:kworker.0 00010000:00010000:3.0:1788202843.841051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202844.865096:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202845.889113:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202846.913049:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202847.937055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202848.129054:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202848.129065:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202848.129069:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202848.129081:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2119 00000100:00080000:1.0:1788202848.129094:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202848.129101:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202848.129164:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442913280/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202848.129204:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202848.129209:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202848.129211:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202848.129217:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2119 00000100:00080000:1.0:1788202848.129222:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202848.129228:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202848.129241:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202848.129243:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442913280:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202848.129263:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442913408/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202848.129272:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202848.129275:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202848.129277:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202848.129280:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2119 00000100:00080000:1.0:1788202848.129284:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202848.129288:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202848.129292:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202848.129302:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442913536/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202848.129308:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202848.129311:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202848.129313:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202848.129315:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2119 00000100:00080000:1.0:1788202848.129317:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788202848.129321:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788202848.129325:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202848.129329:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202848.129338:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442913664/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202848.129380:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202848/real 1788202848] req@ffff90234519ea00 x1875064442913280/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202848.129404:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442913280/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202848.129413:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202848.129416:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202848.129419:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442913280:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202848.129444:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442913408:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202848.129463:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202848.129472:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202848/real 1788202848] req@ffff90234519c700 x1875064442913408/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202848.129481:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442913408/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202848.129491:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202848.129493:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202848.129496:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442913408:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202848.129500:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442913536:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202848.129512:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202848.129518:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202848/real 1788202848] req@ffff90234519fb80 x1875064442913536/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202848.129526:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442913536/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202848.129531:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202848.129533:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202848.129536:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442913536:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202848.129542:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442913664:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202848.129562:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202848.129573:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202848/real 1788202848] req@ffff90234519df80 x1875064442913664/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202848.129587:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442913664/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202848.129595:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202848.129597:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202848.129600:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442913664:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202848.961087:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202849.985095:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202851.009060:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202852.033004:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202853.057094:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202853.249039:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202853.249051:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202853.249055:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202853.249066:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2124 00000100:00080000:1.0:1788202853.249079:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202853.249088:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202853.249138:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442913792/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202853.249168:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202853.249172:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202853.249174:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202853.249178:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2124 00000100:00080000:1.0:1788202853.249182:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202853.249188:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202853.249201:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202853.249216:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442913792:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202853.249221:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442913920/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202853.249235:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202853.249239:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202853.249241:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202853.249246:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2124 00000100:00080000:1.0:1788202853.249250:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202853.249254:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202853.249259:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202853.249271:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442914048/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202853.249281:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202853.249285:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202853.249287:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202853.249291:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2124 00000800:00000100:2.0:1788202853.249295:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788202853.249299:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202853.249306:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202853.249311:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202853.249325:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442914176/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202853.249351:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202853/real 1788202853] req@ffff90234519ed80 x1875064442913792/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202853.249379:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442913792/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202853.249392:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202853.249396:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202853.249402:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442913792:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202853.249432:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442913920:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202853.249459:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202853.249473:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202853/real 1788202853] req@ffff90234519f800 x1875064442913920/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202853.249482:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442913920/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202853.249488:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202853.249490:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202853.249493:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442913920:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202853.249497:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442914048:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202853.249513:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202853.249520:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202853/real 1788202853] req@ffff90234519d880 x1875064442914048/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202853.249528:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442914048/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202853.249534:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202853.249536:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202853.249537:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442914048:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202853.249541:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442914176:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202853.249551:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202853.249556:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202853/real 1788202853] req@ffff90234519c000 x1875064442914176/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202853.249564:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442914176/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202853.249568:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202853.249570:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202853.249572:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442914176:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202854.081099:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202855.105071:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202856.129080:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202857.153011:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202858.177111:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202858.368994:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202858.369002:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202858.369006:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202858.369014:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2129 00000100:00080000:1.0:1788202858.369023:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202858.369028:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202858.369064:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442914304/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202858.369097:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202858.369101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202858.369103:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202858.369108:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2129 00000100:00080000:1.0:1788202858.369112:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202858.369115:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202858.369123:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202858.369135:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442914432/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202858.369146:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202858.369151:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788202858.369151:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442914304:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202858.369153:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202858.369163:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2129 00000100:00080000:1.0:1788202858.369166:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202858.369168:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202858.369172:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202858.369183:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442914560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202858.369190:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202858.369193:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202858.369194:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202858.369197:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2129 00000100:00080000:1.0:1788202858.369200:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202858.369203:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202858.369206:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202858.369216:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202858.369217:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442914688/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202858.369262:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202858/real 1788202858] req@ffff90234519e300 x1875064442914304/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202858.369282:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442914304/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202858.369291:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202858.369294:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202858.369297:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442914304:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202858.369316:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442914432:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202858.369331:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202858.369348:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202858/real 1788202858] req@ffff90234519c000 x1875064442914432/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202858.369357:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442914432/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202858.369363:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202858.369365:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202858.369367:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442914432:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202858.369373:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442914560:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202858.369385:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202858.369392:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202858/real 1788202858] req@ffff90234519d880 x1875064442914560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202858.369401:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442914560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202858.369410:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202858.369413:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202858.369416:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442914560:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202858.369422:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442914688:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202858.369443:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202858.369455:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202858/real 1788202858] req@ffff90234519f800 x1875064442914688/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202858.369466:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442914688/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202858.369471:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202858.369474:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202858.369476:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442914688:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202859.201061:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202860.225065:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202861.249053:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202862.273097:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202863.297052:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202863.489033:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202863.489044:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202863.489049:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202863.489060:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2135 00000100:00080000:3.0:1788202863.489071:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202863.489078:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202863.489114:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442914816/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202863.489145:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202863.489149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202863.489151:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202863.489156:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2135 00000100:00080000:3.0:1788202863.489159:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202863.489163:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202863.489174:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202863.489188:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442914944/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202863.489201:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442914816:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202863.489203:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202863.489212:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202863.489215:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202863.489219:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2135 00000100:00080000:3.0:1788202863.489222:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202863.489226:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202863.489231:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202863.489243:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442915072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202863.489258:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202863.489270:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202863.489273:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788202863.489278:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788202863.489282:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2135 00000100:00080000:3.0:1788202863.489288:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202863.489292:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202863.489298:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202863.489311:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442915200/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202863.489322:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202863/real 1788202863] req@ffff902352688e00 x1875064442914816/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202863.489346:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442914816/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202863.489355:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202863.489358:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202863.489361:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442914816:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202863.489382:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442914944:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202863.489399:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202863.489418:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202863/real 1788202863] req@ffff902352689c00 x1875064442914944/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202863.489428:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442914944/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202863.489434:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202863.489436:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202863.489439:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442914944:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202863.489443:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442915072:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202863.489455:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202863.489462:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202863/real 1788202863] req@ffff90235268a680 x1875064442915072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202863.489470:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442915072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202863.489480:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202863.489483:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202863.489485:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442915072:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202863.489488:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442915200:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202863.489498:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202863.489503:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202863/real 1788202863] req@ffff902352689500 x1875064442915200/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202863.489510:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442915200/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202863.489516:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202863.489518:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202863.489520:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442915200:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202864.321090:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202865.345126:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202866.369059:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202867.393037:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202868.417057:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202868.608973:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202868.608984:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202868.608988:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202868.608999:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2140 00000100:00080000:1.0:1788202868.609011:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202868.609019:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202868.609055:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442915328/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202868.609084:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202868.609088:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202868.609090:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202868.609094:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2140 00000100:00080000:1.0:1788202868.609097:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202868.609101:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202868.609111:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202868.609125:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442915456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202868.609130:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442915328:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202868.609145:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202868.609149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202868.609151:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202868.609156:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2140 00000100:00080000:1.0:1788202868.609160:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202868.609165:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202868.609171:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202868.609183:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442915584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202868.609194:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202868.609197:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202868.609199:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202868.609203:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2140 00000800:00000100:2.0:1788202868.609208:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788202868.609209:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202868.609226:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202868.609232:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202868.609245:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442915712/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202868.609255:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202868/real 1788202868] req@ffff90234519ed80 x1875064442915328/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202868.609271:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442915328/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202868.609279:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202868.609283:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202868.609286:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442915328:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202868.609306:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442915456:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202868.609333:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202868.609343:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202868/real 1788202868] req@ffff90234519df80 x1875064442915456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202868.609352:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442915456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202868.609358:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202868.609360:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202868.609362:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442915456:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202868.609367:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442915584:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202868.609379:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202868.609386:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202868/real 1788202868] req@ffff90234519fb80 x1875064442915584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202868.609393:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442915584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202868.609399:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202868.609400:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202868.609402:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442915584:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202868.609407:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442915712:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202868.609424:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202868.609436:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202868/real 1788202868] req@ffff90234519c700 x1875064442915712/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202868.609450:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442915712/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202868.609459:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202868.609463:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202868.609466:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442915712:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202869.441061:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202870.465090:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202871.489044:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202872.513141:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202873.537098:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202873.729094:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202873.729102:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202873.729107:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202873.729120:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2145 00000100:00080000:1.0:1788202873.729134:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202873.729142:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202873.729187:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442915840/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202873.729217:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202873.729220:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202873.729221:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202873.729226:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2145 00000100:00080000:1.0:1788202873.729228:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202873.729232:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202873.729240:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202873.729253:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442915968/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202873.729260:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202873.729263:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202873.729265:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202873.729269:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2145 00000100:00080000:1.0:1788202873.729272:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788202873.729278:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442915840:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202873.729280:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202873.729288:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202873.729307:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442916096/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202873.729326:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202873.729330:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202873.729332:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202873.729337:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2145 00000100:00080000:1.0:1788202873.729341:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202873.729351:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202873.729358:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202873.729370:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202873.729371:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442916224/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202873.729509:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202873/real 1788202873] req@ffff90234519ea00 x1875064442915840/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202873.729530:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442915840/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202873.729542:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202873.729546:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202873.729552:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442915840:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202873.729563:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442915968:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788202873.729589:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202873.729720:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202873/real 1788202873] req@ffff90234519c380 x1875064442915968/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202873.729735:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442915968/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202873.729743:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202873.729747:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202873.729751:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442915968:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202873.729837:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442916096:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202873.729858:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202873.729868:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202873/real 1788202873] req@ffff90234519f480 x1875064442916096/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202873.729910:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442916096/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202873.729919:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202873.729923:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202873.729928:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442916096:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202873.729936:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442916224:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788202873.729956:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202873.729966:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202873/real 1788202873] req@ffff90234519dc00 x1875064442916224/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202873.729977:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442916224/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202873.729990:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202873.729993:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202873.729996:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442916224:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202874.561029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202875.585122:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202876.609104:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202877.633105:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202878.657120:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202878.849028:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202878.849044:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202878.849049:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202878.849063:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2150 00000100:00080000:3.0:1788202878.849078:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202878.849085:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202878.849131:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442916352/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202878.849160:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202878.849164:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202878.849165:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202878.849168:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2150 00000100:00080000:3.0:1788202878.849170:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202878.849173:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202878.849182:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202878.849192:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442916480/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202878.849199:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202878.849202:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202878.849204:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202878.849207:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2150 00000100:00080000:3.0:1788202878.849209:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202878.849212:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202878.849215:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202878.849222:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442916608/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202878.849231:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202878.849236:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202878.849238:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202878.849242:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2150 00000100:00080000:3.0:1788202878.849245:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788202878.849246:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442916352:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202878.849250:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202878.849258:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202878.849271:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442916736/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202878.849340:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202878.849388:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202878/real 1788202878] req@ffff902352689180 x1875064442916352/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202878.849446:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442916352/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202878.849459:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202878.849464:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202878.849469:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442916352:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202878.849478:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442916480:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202878.849505:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202878.849520:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202878/real 1788202878] req@ffff902352688700 x1875064442916480/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202878.849533:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442916480/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202878.849550:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202878.849553:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202878.849557:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442916480:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202878.849585:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442916608:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202878.849608:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202878.849621:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202878/real 1788202878] req@ffff90235268bb80 x1875064442916608/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202878.849633:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442916608/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202878.849642:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202878.849646:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202878.849650:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442916608:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202878.849656:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442916736:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202878.849679:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202878.849691:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202878/real 1788202878] req@ffff902352689880 x1875064442916736/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202878.849705:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442916736/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202878.849714:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202878.849718:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202878.849722:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442916736:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202879.681082:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202880.705066:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202881.729041:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202882.753067:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202883.777064:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202883.969037:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202883.969049:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202883.969053:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202883.969062:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2155 00000100:00080000:1.0:1788202883.969072:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202883.969078:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202883.969112:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442916864/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202883.969143:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202883.969146:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202883.969148:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202883.969151:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2155 00000100:00080000:1.0:1788202883.969153:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202883.969155:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202883.969163:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202883.969174:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442916992/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202883.969181:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202883.969184:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202883.969186:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202883.969191:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2155 00000100:00100000:2.0:1788202883.969193:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442916864:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202883.969194:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202883.969200:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202883.969207:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202883.969215:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442917120/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202883.969221:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202883.969224:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202883.969225:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202883.969227:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2155 00000100:00080000:1.0:1788202883.969230:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202883.969232:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202883.969235:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202883.969244:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442917248/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202883.969282:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202883.969335:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202883/real 1788202883] req@ffff90234519f100 x1875064442916864/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202883.969356:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442916864/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202883.969367:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202883.969372:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202883.969377:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442916864:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202883.969402:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442916992:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202883.969436:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202883.969452:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202883/real 1788202883] req@ffff90234519dc00 x1875064442916992/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202883.969475:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442916992/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202883.969485:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202883.969489:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202883.969493:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442916992:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202883.969500:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442917120:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202883.969521:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202883.969534:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202883/real 1788202883] req@ffff90234519f480 x1875064442917120/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202883.969548:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442917120/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202883.969558:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202883.969561:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202883.969565:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442917120:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202883.969571:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442917248:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202883.969589:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202883.969601:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202883/real 1788202883] req@ffff90234519c380 x1875064442917248/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202883.969614:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442917248/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202883.969622:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202883.969626:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202883.969630:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442917248:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202884.801032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202885.825063:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202886.849116:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202887.873039:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202888.897058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202889.089012:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202889.089023:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202889.089027:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202889.089038:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2160 00000100:00080000:3.0:1788202889.089051:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202889.089059:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202889.089099:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442917376/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202889.089132:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202889.089136:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202889.089138:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202889.089144:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2160 00000100:00080000:3.0:1788202889.089148:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202889.089153:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202889.089163:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202889.089176:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442917504/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202889.089184:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202889.089187:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202889.089189:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202889.089191:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2160 00000100:00080000:3.0:1788202889.089194:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788202889.089199:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442917376:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202889.089199:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202889.089207:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202889.089214:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442917632/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202889.089222:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202889.089240:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202889.089241:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202889.089249:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2160 00000100:00080000:3.0:1788202889.089253:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202889.089256:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202889.089259:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202889.089267:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442917760/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202889.089327:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202889.089368:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202889/real 1788202889] req@ffff902352688380 x1875064442917376/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202889.089390:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442917376/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202889.089402:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202889.089407:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202889.089411:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442917376:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202889.089420:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442917504:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202889.089445:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202889.089459:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202889/real 1788202889] req@ffff90235268ad80 x1875064442917504/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202889.089473:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442917504/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202889.089481:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202889.089485:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202889.089489:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442917504:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202889.089515:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442917632:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202889.089538:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202889.089551:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202889/real 1788202889] req@ffff90235268b480 x1875064442917632/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202889.089565:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442917632/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202889.089573:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202889.089577:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202889.089580:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442917632:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202889.089587:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442917760:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202889.089608:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202889.089620:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202889/real 1788202889] req@ffff902352688a80 x1875064442917760/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202889.089633:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442917760/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202889.089642:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202889.089646:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202889.089649:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442917760:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202889.921048:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202890.945048:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202891.969046:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202892.993048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202894.017045:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202894.209032:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202894.209040:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202894.209043:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202894.209052:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2165 00000100:00080000:1.0:1788202894.209061:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202894.209067:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202894.209120:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442917888/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202894.209170:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202894.209174:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202894.209175:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202894.209179:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2165 00000100:00080000:1.0:1788202894.209182:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202894.209185:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202894.209194:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202894.209206:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442918016/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202894.209216:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442917888:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202894.209217:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202894.209226:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202894.209227:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202894.209231:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2165 00000100:00080000:1.0:1788202894.209233:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202894.209235:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202894.209239:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202894.209248:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442918144/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202894.209254:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202894.209257:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202894.209259:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202894.209261:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2165 00000100:00080000:1.0:1788202894.209264:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202894.209267:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202894.209270:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202894.209277:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442918272/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202894.209288:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202894.209334:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202894/real 1788202894] req@ffff90234519ea00 x1875064442917888/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202894.209351:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442917888/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202894.209359:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202894.209362:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202894.209365:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442917888:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202894.209394:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442918016:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202894.209419:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202894.209442:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202894/real 1788202894] req@ffff90234519c700 x1875064442918016/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202894.209455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442918016/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202894.209465:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202894.209468:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202894.209471:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442918016:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202894.209478:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442918144:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202894.209501:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202894.209524:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202894/real 1788202894] req@ffff90234519fb80 x1875064442918144/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202894.209539:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442918144/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202894.209548:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202894.209552:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202894.209556:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442918144:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202894.209562:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442918272:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202894.209584:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202894.209597:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202894/real 1788202894] req@ffff90234519df80 x1875064442918272/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202894.209609:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442918272/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202894.209624:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202894.209628:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202894.209632:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442918272:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202895.041065:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202896.065081:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202897.089037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202898.113082:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202899.137070:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202899.328993:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202899.329002:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202899.329005:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202899.329014:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2170 00000100:00080000:1.0:1788202899.329033:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202899.329039:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202899.329097:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442918400/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202899.329135:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202899.329139:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202899.329141:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202899.329146:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2170 00000100:00080000:1.0:1788202899.329150:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202899.329155:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202899.329165:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202899.329182:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442918400:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202899.329187:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442918528/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202899.329197:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202899.329200:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202899.329201:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202899.329204:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2170 00000100:00080000:1.0:1788202899.329207:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202899.329210:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202899.329214:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202899.329223:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442918656/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202899.329232:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202899.329237:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202899.329240:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788202899.329241:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788202899.329249:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2170 00000100:00080000:1.0:1788202899.329255:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202899.329260:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202899.329267:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202899.329281:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442918784/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202899.329286:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202899/real 1788202899] req@ffff90234519ed80 x1875064442918400/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202899.329313:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442918400/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202899.329325:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202899.329329:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202899.329332:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442918400:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202899.329353:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442918528:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202899.329371:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202899.329380:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202899/real 1788202899] req@ffff90234519f800 x1875064442918528/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202899.329389:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442918528/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202899.329395:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202899.329397:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202899.329400:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442918528:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202899.329405:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442918656:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202899.329415:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202899.329422:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202899/real 1788202899] req@ffff90234519d880 x1875064442918656/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202899.329430:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442918656/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202899.329435:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202899.329437:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202899.329439:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442918656:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202899.329442:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442918784:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202899.329451:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202899.329456:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202899/real 1788202899] req@ffff90234519c000 x1875064442918784/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202899.329464:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442918784/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202899.329472:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202899.329476:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202899.329480:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442918784:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202900.161054:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202901.185079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202902.209041:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202903.233097:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202904.257074:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202904.449024:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202904.449035:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202904.449040:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202904.449051:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2175 00000100:00080000:1.0:1788202904.449063:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202904.449070:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202904.449122:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442918912/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202904.449155:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202904.449158:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202904.449160:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202904.449163:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2175 00000100:00080000:1.0:1788202904.449167:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202904.449170:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202904.449179:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202904.449191:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442919040/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202904.449199:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202904.449202:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202904.449203:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202904.449206:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2175 00000100:00080000:1.0:1788202904.449208:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202904.449211:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202904.449214:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202904.449223:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442919168/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202904.449231:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202904.449235:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202904.449242:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202904.449243:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442918912:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202904.449249:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2175 00000100:00080000:1.0:1788202904.449259:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202904.449264:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202904.449266:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202904.449276:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442919296/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202904.449353:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442919040:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202904.449388:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442919168:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202904.449403:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442919296:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202905.281164:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202906.305027:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202907.329082:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202907.521172:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:3.0:1788202907.521238:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202907.521260:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:3.0:1788202907.521308:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:3.0:1788202907.521330:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:3.0:1788202907.521342:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:2.0:1788202907.521369:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202904/real 1788202907] req@ffff90234519d880 x1875064442919168/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202907.521398:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442919168/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202907.521411:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202907.521418:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202907.521422:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442919168:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202907.521431:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202904/real 1788202907] req@ffff90234519f800 x1875064442919296/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202907.521445:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442919296/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202907.521454:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202907.521457:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202907.521460:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442919296:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202907.521468:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202904/real 1788202907] req@ffff90234519e300 x1875064442918912/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202907.521483:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442918912/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202907.521498:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202907.521502:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202907.521505:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442918912:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202907.521513:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202904/real 1788202907] req@ffff90234519c000 x1875064442919040/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202907.521526:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442919040/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202907.521534:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202907.521538:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202907.521541:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442919040:192.168.201.130@tcp:38:kworker.0 00010000:00010000:0.0:1788202908.353014:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202909.377062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202909.569051:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202909.569060:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202909.569063:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202909.569075:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2181 00000100:00080000:3.0:1788202909.569086:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202909.569093:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202909.569138:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064442919424/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202909.569174:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202909.569177:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202909.569179:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202909.569186:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2181 00000100:00080000:3.0:1788202909.569190:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202909.569196:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202909.569206:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442919424:192.168.201.130@tcp:250:kworker.0 00000008:00100000:3.0:1788202909.569209:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202909.569231:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442919552/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202909.569243:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202909.569247:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202909.569250:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202909.569254:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2181 00000100:00080000:3.0:1788202909.569259:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202909.569264:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202909.569270:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202909.569285:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442919680/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202909.569295:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202909.569300:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202909.569304:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788202909.569305:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788202909.569314:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2181 00000100:00080000:3.0:1788202909.569319:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202909.569325:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202909.569330:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202909.569344:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442919808/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202909.569376:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202909/real 1788202909] req@ffff90235268aa00 x1875064442919424/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202909.569404:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064442919424/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202909.569419:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202909.569424:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202909.569431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442919424:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202909.569482:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442919552:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202909.569510:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202909.569523:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202909/real 1788202909] req@ffff902352688a80 x1875064442919552/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202909.569535:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442919552/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202909.569543:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202909.569546:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202909.569549:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442919552:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202909.569555:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442919680:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202909.569575:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202909.569585:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202909/real 1788202909] req@ffff90235268b480 x1875064442919680/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202909.569595:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442919680/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202909.569604:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202909.569607:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202909.569610:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442919680:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202909.569617:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442919808:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202909.569635:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202909.569646:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202909/real 1788202909] req@ffff90235268ad80 x1875064442919808/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202909.569664:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442919808/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202909.569671:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202909.569674:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202909.569677:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442919808:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202910.401024:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202911.425026:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202912.449047:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202913.473077:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202914.497041:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202914.688981:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202914.688990:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202914.688993:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202914.689002:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2186 00000100:00080000:3.0:1788202914.689011:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202914.689017:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202914.689057:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442919936/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202914.689087:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202914.689091:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202914.689093:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202914.689098:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2186 00000100:00080000:3.0:1788202914.689102:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202914.689105:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202914.689114:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202914.689125:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442920064/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202914.689134:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202914.689136:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202914.689140:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788202914.689141:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442919936:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202914.689145:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2186 00000100:00080000:3.0:1788202914.689152:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202914.689154:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202914.689158:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202914.689167:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442920192/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202914.689174:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202914.689177:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202914.689178:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202914.689181:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2186 00000100:00080000:3.0:1788202914.689184:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202914.689186:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202914.689189:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202914.689195:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442920320/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202914.689208:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202914.689261:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202914/real 1788202914] req@ffff902352688380 x1875064442919936/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202914.689282:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442919936/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202914.689294:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202914.689298:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202914.689311:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442919936:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788202914.689336:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442920064:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788202914.689356:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202914.689373:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202914/real 1788202914] req@ffff902352689880 x1875064442920064/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202914.689381:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442920064/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202914.689387:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202914.689390:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202914.689392:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442920064:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788202914.689396:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442920192:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202914.689408:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202914.689414:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202914/real 1788202914] req@ffff90235268bb80 x1875064442920192/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202914.689421:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442920192/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202914.689427:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202914.689429:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202914.689431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442920192:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788202914.689434:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442920320:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788202914.689444:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788202914.689450:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202914/real 1788202914] req@ffff902352688700 x1875064442920320/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202914.689457:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442920320/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202914.689462:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202914.689463:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788202914.689466:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442920320:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202915.521085:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202916.545097:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202917.569049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202918.593034:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202919.617037:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202919.809005:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202919.809013:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202919.809016:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202919.809025:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2191 00000100:00080000:0.0:1788202919.809035:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202919.809042:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202919.809076:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442920448/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202919.809100:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202919.809103:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202919.809104:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202919.809107:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2191 00000100:00080000:0.0:1788202919.809110:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202919.809114:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202919.809122:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202919.809136:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442920576/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202919.809142:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442920448:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202919.809151:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202919.809156:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202919.809158:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202919.809162:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2191 00000100:00080000:0.0:1788202919.809167:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202919.809171:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202919.809177:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202919.809188:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442920704/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202919.809198:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788202919.809201:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788202919.809206:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202919.809209:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202919.809214:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2191 00000100:00080000:0.0:1788202919.809219:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202919.809223:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202919.809227:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202919.809240:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202919/real 1788202919] req@ffff9023450ba680 x1875064442920448/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202919.809243:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442920832/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202919.809258:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442920448/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202919.809280:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202919.809285:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202919.809290:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442920448:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202919.809316:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442920576:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202919.809339:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202919.809348:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202919/real 1788202919] req@ffff9023450b8e00 x1875064442920576/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202919.809356:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442920576/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202919.809362:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202919.809364:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202919.809366:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442920576:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202919.809369:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442920704:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202919.809381:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202919.809386:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202919/real 1788202919] req@ffff9023450bbb80 x1875064442920704/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202919.809393:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442920704/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202919.809398:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202919.809399:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202919.809401:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442920704:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202919.809404:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442920832:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202919.809420:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202919.809426:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202919/real 1788202919] req@ffff9023450bb480 x1875064442920832/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202919.809437:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442920832/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202919.809442:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202919.809443:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202919.809445:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442920832:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202920.641006:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202921.665035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202922.689027:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202923.713071:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202924.737052:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202924.928979:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202924.928990:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202924.928994:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202924.929003:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2196 00000100:00080000:0.0:1788202924.929015:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202924.929024:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202924.929074:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442920960/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202924.929100:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202924.929103:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202924.929104:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202924.929107:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2196 00000100:00080000:0.0:1788202924.929109:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202924.929112:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202924.929122:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202924.929132:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442921088/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202924.929138:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202924.929141:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202924.929142:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202924.929144:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2196 00000100:00080000:0.0:1788202924.929146:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202924.929150:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202924.929151:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442920960:192.168.201.130@tcp:250:kworker.0 00000008:00100000:0.0:1788202924.929153:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202924.929163:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442921216/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202924.929169:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202924.929172:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202924.929173:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202924.929176:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2196 00000100:00080000:0.0:1788202924.929179:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202924.929181:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202924.929185:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202924.929193:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442921344/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202924.929226:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202924.929270:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202924/real 1788202924] req@ffff9023450bb480 x1875064442920960/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202924.929290:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442920960/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202924.929298:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202924.929300:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202924.929303:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442920960:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202924.929311:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442921088:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202924.929335:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202924.929348:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202924/real 1788202924] req@ffff9023450bbb80 x1875064442921088/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202924.929358:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442921088/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202924.929375:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202924.929377:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202924.929381:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442921088:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202924.929407:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442921216:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202924.929433:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202924.929444:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202924/real 1788202924] req@ffff9023450b8e00 x1875064442921216/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202924.929454:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442921216/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202924.929460:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202924.929463:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202924.929466:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442921216:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202924.929471:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442921344:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202924.929492:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202924.929502:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202924/real 1788202924] req@ffff9023450ba680 x1875064442921344/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202924.929512:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442921344/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202924.929518:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202924.929521:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202924.929525:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442921344:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202925.760988:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202926.785070:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202927.809057:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202928.833057:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202929.857145:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202930.049020:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202930.049031:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202930.049036:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202930.049048:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2201 00000100:00080000:0.0:1788202930.049063:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202930.049074:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788202930.049124:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442921472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202930.049155:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202930.049158:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202930.049160:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202930.049164:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2201 00000100:00080000:0.0:1788202930.049167:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202930.049170:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202930.049181:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202930.049191:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442921600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202930.049200:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202930.049202:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202930.049204:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202930.049208:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2201 00000100:00100000:2.0:1788202930.049210:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442921472:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788202930.049211:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202930.049216:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202930.049221:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202930.049227:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442921728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202930.049234:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202930.049237:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202930.049238:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202930.049240:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2201 00000100:00080000:0.0:1788202930.049242:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202930.049244:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788202930.049247:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202930.049256:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442921856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202930.049288:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202930.049337:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202930/real 1788202930] req@ffff9023450b8380 x1875064442921472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202930.049353:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442921472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202930.049360:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202930.049363:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202930.049365:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442921472:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202930.049386:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442921600:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202930.049399:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202930.049407:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202930/real 1788202930] req@ffff9023450b8a80 x1875064442921600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202930.049413:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442921600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202930.049418:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202930.049420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202930.049421:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442921600:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202930.049425:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442921728:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202930.049435:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202930.049442:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202930/real 1788202930] req@ffff9023450bb800 x1875064442921728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202930.049455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442921728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202930.049460:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202930.049462:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202930.049464:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442921728:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202930.049467:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442921856:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202930.049478:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202930.049488:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202930/real 1788202930] req@ffff9023450b9500 x1875064442921856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202930.049499:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442921856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202930.049506:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202930.049509:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202930.049537:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442921856:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202930.881057:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202931.905082:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202932.929039:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202933.953027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202934.977013:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202935.168986:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202935.168996:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202935.169000:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202935.169018:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2206 00000100:00080000:1.0:1788202935.169029:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202935.169036:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202935.169070:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442921984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202935.169092:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202935.169095:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202935.169096:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202935.169099:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2206 00000100:00080000:1.0:1788202935.169102:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202935.169104:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202935.169111:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202935.169119:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442922112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202935.169126:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202935.169128:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202935.169129:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202935.169132:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2206 00000100:00080000:1.0:1788202935.169133:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202935.169137:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202935.169138:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442921984:192.168.201.130@tcp:250:kworker.0 00000008:00100000:1.0:1788202935.169140:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202935.169155:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442922240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202935.169162:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202935.169164:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202935.169165:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202935.169168:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2206 00000100:00080000:1.0:1788202935.169170:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202935.169172:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202935.169175:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202935.169181:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442922368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202935.169200:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202935.169236:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202935/real 1788202935] req@ffff90234519ed80 x1875064442921984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202935.169253:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442921984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202935.169265:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202935.169269:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202935.169277:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442921984:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202935.169286:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442922112:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202935.169308:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202935.169324:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202935/real 1788202935] req@ffff90234519df80 x1875064442922112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202935.169333:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442922112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202935.169338:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202935.169340:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202935.169342:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442922112:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202935.169370:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442922240:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202935.169388:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202935.169399:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202935/real 1788202935] req@ffff90234519fb80 x1875064442922240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202935.169410:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442922240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202935.169418:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202935.169420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202935.169431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442922240:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202935.169438:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442922368:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202935.169452:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202935.169459:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202935/real 1788202935] req@ffff90234519c700 x1875064442922368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202935.169466:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442922368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202935.169471:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202935.169473:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202935.169475:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442922368:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202936.000994:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202937.025095:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202938.049083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202939.073067:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202940.097011:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202940.288962:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202940.288971:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202940.288975:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202940.288982:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2211 00000100:00080000:1.0:1788202940.288993:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202940.288998:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202940.289030:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442922496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202940.289059:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202940.289062:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202940.289063:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202940.289068:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2211 00000100:00080000:1.0:1788202940.289070:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202940.289073:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202940.289080:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202940.289089:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442922624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202940.289097:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202940.289100:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202940.289101:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202940.289104:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2211 00000100:00080000:1.0:1788202940.289106:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202940.289109:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202940.289112:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202940.289117:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442922752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202940.289124:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202940.289127:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202940.289128:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202940.289134:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442922496:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202940.289135:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2211 00000100:00080000:1.0:1788202940.289138:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202940.289143:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202940.289149:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202940.289156:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442922880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202940.289208:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202940.289246:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202940/real 1788202940] req@ffff90234519ea00 x1875064442922496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202940.289265:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442922496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202940.289275:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202940.289279:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202940.289283:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442922496:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202940.289291:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442922624:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202940.289311:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202940.289322:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202940/real 1788202940] req@ffff90234519c380 x1875064442922624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202940.289352:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442922624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202940.289362:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202940.289366:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202940.289369:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442922624:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202940.289394:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442922752:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202940.289416:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202940.289428:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202940/real 1788202940] req@ffff90234519f480 x1875064442922752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202940.289440:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442922752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202940.289449:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202940.289453:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202940.289456:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442922752:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202940.289462:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442922880:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202940.289479:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202940.289489:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202940/real 1788202940] req@ffff90234519dc00 x1875064442922880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202940.289500:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442922880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202940.289515:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202940.289518:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202940.289522:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442922880:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202941.121100:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202942.145053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202943.169145:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202944.193106:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202945.217041:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202945.408975:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202945.408983:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202945.408986:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202945.408995:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2216 00000100:00080000:3.0:1788202945.409004:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202945.409010:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202945.409043:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442923008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202945.409066:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202945.409070:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202945.409072:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202945.409076:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2216 00000100:00080000:3.0:1788202945.409080:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202945.409084:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202945.409094:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202945.409107:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442923136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202945.409113:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442923008:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202945.409118:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202945.409121:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202945.409123:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202945.409127:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2216 00000100:00080000:3.0:1788202945.409130:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202945.409133:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202945.409137:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202945.409145:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442923264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202945.409153:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202945.409156:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202945.409158:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202945.409161:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2216 00000100:00080000:3.0:1788202945.409163:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202945.409167:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202945.409171:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788202945.409173:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202945.409179:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442923392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202945.409211:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202945/real 1788202945] req@ffff902352689180 x1875064442923008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202945.409229:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442923008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202945.409239:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202945.409242:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202945.409275:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442923008:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202945.409299:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442923136:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202945.409322:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202945.409337:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202945/real 1788202945] req@ffff902352689500 x1875064442923136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202945.409351:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442923136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202945.409361:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202945.409364:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202945.409368:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442923136:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202945.409376:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442923264:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202945.409401:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202945.409411:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202945/real 1788202945] req@ffff90235268a680 x1875064442923264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202945.409425:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442923264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202945.409433:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202945.409437:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202945.409442:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442923264:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202945.409448:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442923392:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202945.409472:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202945.409486:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202945/real 1788202945] req@ffff902352689c00 x1875064442923392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202945.409497:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442923392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202945.409506:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202945.409510:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202945.409513:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442923392:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202946.241052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202947.265071:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202948.289023:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202949.313060:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202950.337040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202950.528973:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202950.528983:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202950.528986:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202950.528995:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2222 00000100:00080000:3.0:1788202950.529005:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202950.529011:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202950.529046:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442923520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202950.529069:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202950.529071:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202950.529073:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202950.529075:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2222 00000100:00080000:3.0:1788202950.529078:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202950.529080:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202950.529088:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202950.529098:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442923648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202950.529106:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442923520:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202950.529109:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202950.529115:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202950.529116:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202950.529120:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2222 00000100:00080000:3.0:1788202950.529121:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202950.529124:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202950.529127:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202950.529136:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442923776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202950.529147:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202950.529150:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202950.529151:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202950.529153:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2222 00000100:00080000:3.0:1788202950.529183:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202950.529186:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202950.529189:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788202950.529195:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202950.529200:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442923904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202950.529237:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202950/real 1788202950] req@ffff902352688e00 x1875064442923520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202950.529252:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442923520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202950.529261:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202950.529264:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202950.529267:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442923520:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202950.529287:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442923648:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202950.529303:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202950.529312:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202950/real 1788202950] req@ffff90235268b100 x1875064442923648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202950.529320:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442923648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202950.529326:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202950.529328:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202950.529330:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442923648:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202950.529334:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442923776:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202950.529345:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202950.529351:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202950/real 1788202950] req@ffff902352688000 x1875064442923776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202950.529359:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442923776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202950.529370:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202950.529372:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202950.529374:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442923776:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202950.529378:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442923904:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202950.529387:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202950.529393:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202950/real 1788202950] req@ffff90235268b800 x1875064442923904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202950.529400:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442923904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202950.529405:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202950.529407:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202950.529409:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442923904:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202951.361067:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202952.385034:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202953.409018:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202954.433032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202955.457082:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202955.648996:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202955.649007:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202955.649012:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202955.649021:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2227 00000100:00080000:1.0:1788202955.649032:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202955.649039:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202955.649115:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442924032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202955.649144:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202955.649148:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202955.649150:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202955.649154:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2227 00000100:00080000:1.0:1788202955.649158:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202955.649162:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202955.649172:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202955.649186:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442924032:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202955.649191:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442924160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202955.649206:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202955.649211:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202955.649213:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202955.649218:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2227 00000100:00080000:1.0:1788202955.649222:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202955.649227:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202955.649233:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788202955.649240:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202955.649255:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442924288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202955.649266:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202955.649270:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202955.649272:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202955.649276:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202955/real 1788202955] req@ffff90234519f100 x1875064442924032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202955.649281:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2227 00000100:00080000:1.0:1788202955.649290:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202955.649296:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202955.649299:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442924032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788202955.649303:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1788202955.649309:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202955.649314:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202955.649320:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442924416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202955.649320:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442924032:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202955.649356:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442924160:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202955.649372:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202955.649387:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202955/real 1788202955] req@ffff90234519dc00 x1875064442924160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202955.649397:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442924160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202955.649403:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202955.649405:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202955.649407:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442924160:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202955.649412:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442924288:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202955.649424:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202955.649430:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202955/real 1788202955] req@ffff90234519f480 x1875064442924288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202955.649438:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442924288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202955.649443:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202955.649445:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202955.649447:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442924288:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202955.649450:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442924416:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202955.649459:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202955.649465:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202955/real 1788202955] req@ffff90234519c380 x1875064442924416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202955.649474:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442924416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202955.649479:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202955.649481:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202955.649483:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442924416:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202956.481082:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202957.505029:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202958.529069:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202959.553026:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202960.577072:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202960.769004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202960.769014:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202960.769018:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202960.769027:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2232 00000100:00080000:1.0:1788202960.769036:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202960.769042:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202960.769075:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442924544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202960.769099:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202960.769102:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202960.769103:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202960.769106:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2232 00000100:00080000:1.0:1788202960.769108:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202960.769110:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202960.769117:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202960.769126:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442924672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202960.769132:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202960.769135:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202960.769136:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202960.769139:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2232 00000100:00080000:1.0:1788202960.769141:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202960.769144:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202960.769147:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202960.769156:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442924800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202960.769156:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442924544:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202960.769172:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202960.769175:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202960.769178:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202960.769182:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2232 00000100:00080000:1.0:1788202960.769186:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202960.769190:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202960.769195:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202960.769208:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442924928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202960.769217:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202960.769255:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202960/real 1788202960] req@ffff90234519ea00 x1875064442924544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202960.769274:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442924544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202960.769285:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202960.769290:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202960.769295:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442924544:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202960.769312:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442924672:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202960.769337:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202960.769351:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202960/real 1788202960] req@ffff90234519c700 x1875064442924672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202960.769366:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442924672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202960.769383:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202960.769387:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202960.769391:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442924672:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202960.769417:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442924800:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202960.769439:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202960.769448:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202960/real 1788202960] req@ffff90234519fb80 x1875064442924800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202960.769456:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442924800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202960.769463:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202960.769465:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202960.769467:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442924800:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202960.769471:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442924928:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202960.769483:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202960.769489:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202960/real 1788202960] req@ffff90234519df80 x1875064442924928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202960.769497:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442924928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202960.769502:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202960.769505:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202960.769509:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442924928:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202961.601059:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202962.625048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202963.649043:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202964.673033:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202965.697034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202965.888986:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202965.888995:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202965.888998:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202965.889006:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2237 00000100:00080000:3.0:1788202965.889016:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202965.889021:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202965.889052:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442925056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202965.889076:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202965.889083:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202965.889084:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202965.889091:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2237 00000100:00080000:3.0:1788202965.889094:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202965.889098:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202965.889112:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202965.889122:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442925056:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202965.889130:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442925184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202965.889145:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202965.889149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202965.889151:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202965.889156:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2237 00000100:00080000:3.0:1788202965.889172:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202965.889175:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202965.889180:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788202965.889181:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788202965.889195:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442925312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202965.889203:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202965.889207:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202965.889209:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202965.889213:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2237 00000100:00080000:3.0:1788202965.889217:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202965.889220:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202965.889226:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202965.889229:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202965/real 1788202965] req@ffff90235268a300 x1875064442925056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202965.889242:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442925440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202965.889248:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442925056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202965.889270:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202965.889274:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202965.889277:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442925056:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202965.889296:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442925184:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202965.889311:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202965.889320:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202965/real 1788202965] req@ffff90235268b800 x1875064442925184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202965.889330:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442925184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202965.889339:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202965.889342:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202965.889346:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442925184:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202965.889352:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442925312:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202965.889376:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202965.889387:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202965/real 1788202965] req@ffff902352688000 x1875064442925312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202965.889395:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442925312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202965.889401:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202965.889404:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202965.889406:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442925312:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202965.889410:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442925440:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202965.889423:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202965.889429:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202965/real 1788202965] req@ffff90235268b100 x1875064442925440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202965.889437:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442925440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202965.889442:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202965.889444:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202965.889446:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442925440:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202966.721050:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202967.745024:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202968.769110:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202969.793143:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202970.817107:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202971.008971:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202971.008986:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202971.008993:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202971.009006:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2242 00000100:00080000:3.0:1788202971.009020:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202971.009030:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202971.009081:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442925568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202971.009117:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202971.009122:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202971.009124:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202971.009130:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2242 00000100:00080000:3.0:1788202971.009135:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202971.009141:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202971.009153:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202971.009175:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442925568:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202971.009175:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442925696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202971.009203:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202971.009208:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202971.009212:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202971.009219:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2242 00000100:00080000:3.0:1788202971.009224:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202971.009229:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202971.009235:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202971.009249:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442925824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202971.009268:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202971.009273:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202971.009276:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202971.009282:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2242 00000100:00080000:3.0:1788202971.009287:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202971.009292:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202971.009299:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442925696:192.168.201.130@tcp:38:kworker.0 00000008:00100000:3.0:1788202971.009300:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202971.009327:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442925952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202971.009338:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442925824:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202971.009386:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442925952:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202971.841061:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202972.865036:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202973.889045:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788202974.081319:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788202974.081378:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788202974.081398:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202974.081446:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202974.081462:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788202974.081481:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788202974.081492:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202971/real 1788202974] req@ffff902352689c00 x1875064442925696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202974.081525:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442925696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202974.081537:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202974.081542:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202974.081546:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442925696:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788202974.081554:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202971/real 1788202974] req@ffff90235268a680 x1875064442925824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202974.081570:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442925824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202974.081580:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202974.081586:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202974.081591:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442925824:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788202974.081600:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202971/real 1788202974] req@ffff902352688e00 x1875064442925568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202974.081614:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442925568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202974.081628:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202974.081632:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202974.081635:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442925568:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202974.081663:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202971/real 1788202974] req@ffff902352689500 x1875064442925952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202974.081675:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442925952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202974.081682:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202974.081685:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788202974.081688:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442925952:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788202974.913004:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202975.937077:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202976.128940:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202976.128949:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202976.128952:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202976.128961:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2247 00000100:00080000:3.0:1788202976.128970:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202976.128975:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202976.129011:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442926080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202976.129051:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202976.129055:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202976.129057:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202976.129060:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2247 00000100:00080000:3.0:1788202976.129063:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202976.129067:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202976.129075:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202976.129086:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442926208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202976.129094:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202976.129097:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202976.129098:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202976.129101:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2247 00000100:00080000:3.0:1788202976.129103:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202976.129105:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202976.129109:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202976.129111:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442926080:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788202976.129119:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442926336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202976.129131:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202976.129135:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202976.129137:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202976.129142:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2247 00000100:00080000:3.0:1788202976.129146:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202976.129150:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202976.129156:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202976.129170:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442926464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202976.129181:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202976.129223:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202976/real 1788202976] req@ffff902352689500 x1875064442926080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202976.129244:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442926080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202976.129255:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202976.129260:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202976.129264:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442926080:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202976.129273:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442926208:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202976.129299:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202976.129313:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202976/real 1788202976] req@ffff902352688e00 x1875064442926208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202976.129328:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442926208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202976.129337:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202976.129341:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202976.129345:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442926208:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202976.129372:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442926336:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202976.129395:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202976.129406:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202976/real 1788202976] req@ffff90235268a680 x1875064442926336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202976.129420:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442926336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202976.129429:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202976.129433:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202976.129436:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442926336:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202976.129443:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442926464:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202976.129463:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202976.129475:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202976/real 1788202976] req@ffff902352689c00 x1875064442926464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202976.129487:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442926464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202976.129495:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202976.129498:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202976.129503:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442926464:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202976.961002:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202977.985043:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202979.009062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202980.033060:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202981.057099:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202981.248974:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202981.248985:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202981.248989:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202981.249000:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2252 00000100:00080000:3.0:1788202981.249016:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202981.249024:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202981.249054:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442926592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202981.249081:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202981.249084:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202981.249086:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202981.249090:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2252 00000100:00080000:3.0:1788202981.249092:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202981.249095:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202981.249102:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202981.249114:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442926720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202981.249124:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788202981.249129:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442926592:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202981.249130:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202981.249136:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202981.249143:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2252 00000100:00080000:3.0:1788202981.249148:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202981.249154:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202981.249160:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202981.249177:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442926848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202981.249187:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202981.249191:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1788202981.249193:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788202981.249194:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202981.249201:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2252 00000100:00080000:3.0:1788202981.249207:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202981.249213:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202981.249219:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202981.249236:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442926976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202981.249237:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202981/real 1788202981] req@ffff902352689180 x1875064442926592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202981.249273:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442926592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202981.249282:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202981.249286:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202981.249289:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442926592:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202981.249309:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442926720:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788202981.249325:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202981.249335:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202981/real 1788202981] req@ffff902352688700 x1875064442926720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202981.249349:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442926720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202981.249358:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202981.249362:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202981.249366:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442926720:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788202981.249373:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442926848:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202981.249395:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202981.249405:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202981/real 1788202981] req@ffff90235268bb80 x1875064442926848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202981.249414:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442926848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202981.249420:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202981.249422:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202981.249424:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442926848:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788202981.249429:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442926976:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788202981.249441:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202981.249447:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202981/real 1788202981] req@ffff902352689880 x1875064442926976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202981.249455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442926976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202981.249460:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202981.249461:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788202981.249463:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442926976:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202982.081039:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202983.105051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202984.129072:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202985.153043:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202986.177069:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202986.369016:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202986.369026:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202986.369030:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202986.369040:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2257 00000100:00080000:1.0:1788202986.369052:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202986.369059:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202986.369098:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442927104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202986.369131:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202986.369135:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202986.369137:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202986.369142:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2257 00000100:00080000:1.0:1788202986.369146:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202986.369152:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202986.369161:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202986.369170:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442927104:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788202986.369182:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442927232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202986.369195:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202986.369198:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202986.369201:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202986.369206:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2257 00000100:00080000:1.0:1788202986.369210:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202986.369215:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202986.369220:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788202986.369231:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788202986.369237:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442927360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202986.369249:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202986.369254:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202986.369256:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202986.369261:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2257 00000100:00080000:1.0:1788202986.369266:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202986.369270:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788202986.369274:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202986/real 1788202986] req@ffff90234519ed80 x1875064442927104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788202986.369278:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202986.369297:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442927488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202986.369306:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442927104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202986.369321:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202986.369325:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202986.369329:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442927104:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202986.369352:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442927232:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202986.369378:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202986.369392:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202986/real 1788202986] req@ffff90234519c000 x1875064442927232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202986.369406:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442927232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202986.369413:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202986.369415:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202986.369417:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442927232:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202986.369422:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442927360:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202986.369445:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202986.369458:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202986/real 1788202986] req@ffff90234519e300 x1875064442927360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202986.369472:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442927360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202986.369481:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202986.369484:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202986.369488:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442927360:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202986.369495:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442927488:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202986.369516:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202986.369529:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202986/real 1788202986] req@ffff90234519f800 x1875064442927488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202986.369577:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442927488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202986.369587:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202986.369590:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202986.369594:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442927488:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788202987.201054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202988.225045:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202989.249008:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202990.273064:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202991.298055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202991.488977:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202991.489018:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202991.489023:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202991.489031:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2263 00000100:00080000:1.0:1788202991.489041:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202991.489046:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788202991.489084:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442927616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202991.489114:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202991.489119:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202991.489121:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202991.489125:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2263 00000100:00080000:1.0:1788202991.489128:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202991.489132:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202991.489140:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202991.489156:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442927744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202991.489165:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442927616:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788202991.489171:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202991.489177:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202991.489179:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202991.489185:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2263 00000100:00080000:1.0:1788202991.489189:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202991.489194:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202991.489199:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202991.489210:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442927872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202991.489223:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788202991.489226:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202991.489229:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202991.489231:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202991.489235:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2263 00000100:00080000:1.0:1788202991.489238:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202991.489241:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788202991.489244:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202991.489252:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442928000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202991.489271:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202991/real 1788202991] req@ffff90234519d880 x1875064442927616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202991.489292:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442927616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202991.489300:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202991.489303:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202991.489306:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442927616:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202991.489323:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442927744:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202991.489345:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202991.489354:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202991/real 1788202991] req@ffff90234519f800 x1875064442927744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202991.489362:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442927744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202991.489368:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202991.489370:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202991.489372:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442927744:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202991.489376:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442927872:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202991.489392:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202991.489400:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202991/real 1788202991] req@ffff90234519e300 x1875064442927872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202991.489407:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442927872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202991.489417:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202991.489419:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202991.489421:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442927872:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202991.489431:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442928000:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202991.489443:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202991.489466:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202991/real 1788202991] req@ffff90234519c000 x1875064442928000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202991.489475:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442928000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202991.489480:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202991.489482:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202991.489484:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442928000:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202992.321089:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202993.345042:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202994.369018:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202995.393074:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202996.417028:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202996.609020:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202996.609028:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202996.609031:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202996.609039:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2268 00000100:00080000:3.0:1788202996.609049:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202996.609055:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788202996.609087:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442928128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202996.609111:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202996.609113:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202996.609115:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202996.609119:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2268 00000100:00080000:3.0:1788202996.609122:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202996.609126:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202996.609136:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202996.609153:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442928256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202996.609158:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442928128:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788202996.609173:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202996.609177:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202996.609179:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202996.609184:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2268 00000100:00080000:3.0:1788202996.609188:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202996.609192:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202996.609199:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202996.609213:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442928384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202996.609222:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788202996.609223:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788202996.609228:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202996.609231:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202996.609234:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2268 00000100:00080000:3.0:1788202996.609239:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202996.609243:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788202996.609253:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202996.609261:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202996/real 1788202996] req@ffff902352688380 x1875064442928128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202996.609266:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442928512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202996.609286:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442928128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202996.609296:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202996.609299:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202996.609302:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442928128:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788202996.609322:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442928256:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788202996.609337:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202996.609376:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202996/real 1788202996] req@ffff90235268ad80 x1875064442928256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202996.609390:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442928256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202996.609399:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202996.609402:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202996.609407:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442928256:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788202996.609414:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442928384:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202996.609434:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202996.609446:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202996/real 1788202996] req@ffff90235268b480 x1875064442928384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202996.609455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442928384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202996.609461:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202996.609463:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202996.609465:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442928384:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788202996.609469:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442928512:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788202996.609482:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788202996.609489:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202996/real 1788202996] req@ffff902352688a80 x1875064442928512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202996.609496:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442928512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202996.609501:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202996.609503:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788202996.609505:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442928512:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788202997.441038:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202998.464999:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202999.489055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203000.513043:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203001.537063:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203001.729030:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203001.729038:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203001.729047:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203001.729055:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2273 00000100:00080000:1.0:1788203001.729064:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203001.729070:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203001.729102:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442928640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203001.729123:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203001.729127:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203001.729128:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203001.729131:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2273 00000100:00080000:1.0:1788203001.729133:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203001.729136:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203001.729148:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203001.729157:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442928768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203001.729166:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203001.729168:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203001.729169:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203001.729172:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2273 00000100:00080000:1.0:1788203001.729174:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203001.729177:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203001.729180:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203001.729184:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442928640:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203001.729190:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442928896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203001.729198:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203001.729200:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203001.729201:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203001.729203:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2273 00000100:00080000:1.0:1788203001.729205:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203001.729208:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203001.729212:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203001.729221:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442929024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203001.729245:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203001.729301:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203001/real 1788203001] req@ffff90234519ed80 x1875064442928640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203001.729315:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442928640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203001.729322:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203001.729325:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203001.729328:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442928640:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203001.729334:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442928768:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203001.729356:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203001.729368:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203001/real 1788203001] req@ffff90234519df80 x1875064442928768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203001.729376:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442928768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203001.729386:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203001.729388:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203001.729390:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442928768:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203001.729408:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442928896:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203001.729420:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203001.729426:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203001/real 1788203001] req@ffff90234519fb80 x1875064442928896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203001.729433:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442928896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203001.729439:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203001.729441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203001.729442:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442928896:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203001.729446:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442929024:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203001.729455:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203001.729461:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203001/real 1788203001] req@ffff90234519c700 x1875064442929024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203001.729468:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442929024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203001.729473:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203001.729474:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203001.729477:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442929024:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203002.561081:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203003.585047:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203004.609040:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203005.633049:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203006.657057:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203006.849020:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203006.849030:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203006.849035:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203006.849045:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2278 00000100:00080000:3.0:1788203006.849057:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203006.849065:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203006.849106:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064442929152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203006.849155:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203006.849160:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203006.849162:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203006.849168:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2278 00000100:00080000:3.0:1788203006.849173:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203006.849197:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203006.849209:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203006.849221:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442929152:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203006.849248:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442929280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203006.849262:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203006.849268:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203006.849270:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203006.849274:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2278 00000100:00080000:3.0:1788203006.849279:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203006.849283:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203006.849288:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203006.849301:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442929408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203006.849306:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788203006.849317:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203006.849321:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203006.849323:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203006.849328:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2278 00000100:00080000:3.0:1788203006.849332:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203006.849337:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203006.849342:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203006.849353:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203006/real 1788203006] req@ffff90235268aa00 x1875064442929152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203006.849357:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442929536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203006.849396:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064442929152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203006.849409:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203006.849413:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203006.849416:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442929152:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203006.849437:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442929280:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203006.849456:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203006.849465:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203006/real 1788203006] req@ffff902352688a80 x1875064442929280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203006.849474:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442929280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203006.849480:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203006.849482:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203006.849485:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442929280:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203006.849489:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442929408:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203006.849504:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203006.849516:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203006/real 1788203006] req@ffff90235268b480 x1875064442929408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203006.849528:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442929408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203006.849538:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203006.849541:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203006.849545:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442929408:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203006.849551:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442929536:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203006.849568:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203006.849577:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203006/real 1788203006] req@ffff90235268ad80 x1875064442929536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203006.849602:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442929536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203006.849610:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203006.849612:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203006.849615:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442929536:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203007.681027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203008.705098:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203009.729044:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203010.753089:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203011.777050:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203011.968967:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203011.968975:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203011.968979:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203011.968987:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2283 00000100:00080000:3.0:1788203011.968997:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203011.969004:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203011.969044:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442929664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203011.969076:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203011.969080:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203011.969082:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203011.969088:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2283 00000100:00080000:3.0:1788203011.969112:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203011.969116:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203011.969124:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203011.969137:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442929664:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203011.969138:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442929792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203011.969163:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203011.969166:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203011.969167:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203011.969172:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2283 00000100:00080000:3.0:1788203011.969175:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203011.969179:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203011.969183:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203011.969194:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442929920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203011.969201:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203011.969205:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203011.969207:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203011.969210:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2283 00000100:00080000:3.0:1788203011.969234:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203011.969237:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203011.969242:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788203011.969247:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203011.969252:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442930048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203011.969300:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203011/real 1788203011] req@ffff902352688380 x1875064442929664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203011.969319:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442929664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203011.969330:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203011.969334:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203011.969338:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442929664:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203011.969364:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442929792:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203011.969384:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203011.969395:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203011/real 1788203011] req@ffff902352689880 x1875064442929792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203011.969405:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442929792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203011.969415:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203011.969420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203011.969424:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442929792:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203011.969432:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442929920:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203011.969457:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203011.969471:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203011/real 1788203011] req@ffff90235268bb80 x1875064442929920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203011.969484:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442929920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203011.969492:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203011.969494:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203011.969497:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442929920:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203011.969503:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442930048:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203011.969520:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203011.969539:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203011/real 1788203011] req@ffff902352688700 x1875064442930048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203011.969548:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442930048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203011.969555:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203011.969557:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203011.969560:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442930048:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203012.801052:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203013.824997:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203014.849052:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203015.873003:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203016.897052:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203017.088955:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203017.088963:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203017.088967:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203017.088975:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2288 00000100:00080000:3.0:1788203017.088984:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203017.088990:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203017.089044:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442930176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203017.089075:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203017.089078:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203017.089080:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203017.089083:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2288 00000100:00080000:3.0:1788203017.089086:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203017.089090:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203017.089099:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203017.089113:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442930304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203017.089123:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788203017.089124:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442930176:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203017.089134:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203017.089137:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203017.089140:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2288 00000100:00080000:3.0:1788203017.089143:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203017.089146:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203017.089149:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203017.089158:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442930432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203017.089165:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203017.089167:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203017.089169:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203017.089171:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2288 00000100:00080000:3.0:1788203017.089173:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203017.089176:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203017.089179:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203017.089189:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442930560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203017.089207:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203017.089254:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203017/real 1788203017] req@ffff902352689180 x1875064442930176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203017.089280:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442930176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203017.089289:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203017.089292:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203017.089295:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442930176:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203017.089314:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442930304:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203017.089332:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203017.089341:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203017/real 1788203017] req@ffff902352689c00 x1875064442930304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203017.089349:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442930304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203017.089354:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203017.089356:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203017.089358:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442930304:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203017.089362:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442930432:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203017.089375:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203017.089381:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203017/real 1788203017] req@ffff90235268a680 x1875064442930432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203017.089388:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442930432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203017.089393:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203017.089395:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203017.089397:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442930432:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203017.089400:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442930560:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203017.089410:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203017.089415:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203017/real 1788203017] req@ffff902352688e00 x1875064442930560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203017.089422:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442930560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203017.089427:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203017.089429:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203017.089431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442930560:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203017.921035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203018.945116:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203019.969094:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203020.993065:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203022.017086:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203022.209007:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203022.209015:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203022.209019:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203022.209027:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2293 00000100:00080000:1.0:1788203022.209037:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203022.209042:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203022.209072:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442930688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203022.209094:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203022.209098:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203022.209100:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203022.209105:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2293 00000100:00080000:1.0:1788203022.209109:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203022.209114:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203022.209124:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203022.209137:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442930688:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203022.209143:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442930816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203022.209153:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203022.209156:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203022.209157:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203022.209161:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2293 00000100:00080000:1.0:1788203022.209163:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203022.209167:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203022.209171:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203022.209188:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442930944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203022.209196:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788203022.209199:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788203022.209200:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203022.209203:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203022.209206:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2293 00000100:00080000:1.0:1788203022.209210:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203022.209213:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203022.209217:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203022.209225:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442931072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203022.209248:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203022/real 1788203022] req@ffff90234519ea00 x1875064442930688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203022.209265:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442930688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203022.209274:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203022.209277:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203022.209280:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442930688:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203022.209298:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442930816:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203022.209313:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203022.209322:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203022/real 1788203022] req@ffff90234519c380 x1875064442930816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203022.209330:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442930816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203022.209335:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203022.209337:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203022.209340:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442930816:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203022.209343:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442930944:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203022.209353:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203022.209359:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203022/real 1788203022] req@ffff90234519f480 x1875064442930944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203022.209366:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442930944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203022.209376:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203022.209378:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203022.209380:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442930944:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203022.209383:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442931072:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203022.209392:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203022.209397:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203022/real 1788203022] req@ffff90234519dc00 x1875064442931072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203022.209404:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442931072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203022.209409:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203022.209411:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203022.209413:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442931072:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203023.041046:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203024.065058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203025.089085:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203026.113019:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203027.137046:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203027.329009:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203027.329019:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203027.329024:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203027.329036:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2298 00000100:00080000:1.0:1788203027.329048:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203027.329057:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203027.329097:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442931200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203027.329125:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203027.329129:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203027.329131:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203027.329135:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2298 00000100:00080000:1.0:1788203027.329140:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203027.329144:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203027.329155:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203027.329165:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442931200:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203027.329174:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442931328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203027.329187:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203027.329191:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203027.329194:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203027.329199:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2298 00000100:00080000:1.0:1788203027.329212:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203027.329218:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:0.0:1788203027.329221:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:1.0:1788203027.329230:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203027.329246:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442931456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203027.329257:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203027.329261:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203027.329263:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788203027.329267:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203027/real 1788203027] req@ffff90234519f100 x1875064442931200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203027.329272:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2298 00000100:00080000:1.0:1788203027.329280:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203027.329287:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203027.329290:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442931200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788203027.329294:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1788203027.329308:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203027.329313:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203027.329313:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442931584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203027.329321:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442931200:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203027.329346:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442931328:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203027.329378:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203027.329393:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203027/real 1788203027] req@ffff90234519dc00 x1875064442931328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203027.329406:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442931328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203027.329415:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203027.329418:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203027.329420:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442931328:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203027.329425:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442931456:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203027.329441:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203027.329449:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203027/real 1788203027] req@ffff90234519f480 x1875064442931456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203027.329456:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442931456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203027.329462:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203027.329464:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203027.329466:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442931456:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203027.329470:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442931584:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203027.329480:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203027.329485:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203027/real 1788203027] req@ffff90234519c380 x1875064442931584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203027.329493:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442931584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203027.329498:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203027.329500:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203027.329502:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442931584:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203028.161003:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203029.185026:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203030.209025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203031.233065:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203032.257110:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203032.449013:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203032.449021:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203032.449025:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203032.449033:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2303 00000100:00080000:3.0:1788203032.449044:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203032.449063:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203032.449101:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442931712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203032.449125:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203032.449128:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203032.449129:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203032.449133:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2303 00000100:00080000:3.0:1788203032.449136:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203032.449139:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203032.449147:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203032.449160:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442931840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203032.449172:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788203032.449175:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442931712:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203032.449178:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203032.449183:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203032.449190:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2303 00000100:00080000:3.0:1788203032.449194:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203032.449199:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203032.449205:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203032.449217:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442931968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203032.449225:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203032.449227:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203032.449228:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203032.449231:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2303 00000800:00000100:0.0:1788203032.449237:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788203032.449238:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203032.449244:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203032.449248:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203032.449257:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442932096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203032.449278:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203032/real 1788203032] req@ffff902352689500 x1875064442931712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203032.449294:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442931712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203032.449302:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203032.449305:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203032.449309:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442931712:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203032.449327:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442931840:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203032.449342:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203032.449351:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203032/real 1788203032] req@ffff90235268b100 x1875064442931840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203032.449366:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442931840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203032.449372:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203032.449375:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203032.449377:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442931840:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203032.449381:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442931968:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203032.449393:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203032.449399:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203032/real 1788203032] req@ffff902352688000 x1875064442931968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203032.449407:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442931968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203032.449412:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203032.449414:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203032.449416:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442931968:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203032.449419:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442932096:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203032.449429:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203032.449435:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203032/real 1788203032] req@ffff90235268b800 x1875064442932096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203032.449447:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442932096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203032.449452:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203032.449453:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203032.449455:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442932096:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203033.281016:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203034.305089:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203035.329057:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203036.353080:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203037.377007:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203037.568974:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203037.568984:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203037.568988:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203037.568998:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2309 00000100:00080000:3.0:1788203037.569009:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203037.569015:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203037.569045:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442932224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203037.569069:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203037.569072:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203037.569073:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203037.569075:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2309 00000100:00080000:3.0:1788203037.569078:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203037.569080:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203037.569087:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203037.569097:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442932352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203037.569103:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203037.569107:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203037.569108:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203037.569111:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2309 00000100:00080000:3.0:1788203037.569113:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788203037.569117:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442932224:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203037.569118:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203037.569124:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203037.569133:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442932480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203037.569139:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203037.569149:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203037.569153:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203037.569155:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2309 00000100:00080000:3.0:1788203037.569158:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203037.569160:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203037.569163:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203037.569170:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442932608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203037.569206:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442932352:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203037.569236:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442932480:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203037.569250:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442932608:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203038.401369:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203039.425815:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203040.448992:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788203040.641164:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788203040.641202:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788203040.641217:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788203040.641258:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788203040.641267:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788203040.641274:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788203040.641369:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203037/real 1788203040] req@ffff902352688000 x1875064442932480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203040.641397:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442932480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203040.641411:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203040.641416:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203040.641421:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442932480:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203040.641428:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203037/real 1788203040] req@ffff90235268b100 x1875064442932608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203040.641440:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442932608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203040.641447:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203040.641450:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203040.641453:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442932608:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203040.641459:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203037/real 1788203040] req@ffff90235268a300 x1875064442932224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203040.641468:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442932224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203040.641474:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203040.641477:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203040.641480:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442932224:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203040.641486:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203037/real 1788203040] req@ffff90235268b800 x1875064442932352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203040.641496:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442932352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203040.641503:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203040.641507:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203040.641509:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442932352:192.168.201.130@tcp:38:kworker.0 00010000:00010000:1.0:1788203041.472980:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203042.497782:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203042.689001:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203042.689015:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203042.689020:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203042.689032:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2314 00000100:00080000:3.0:1788203042.689045:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203042.689052:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203042.689092:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442932736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203042.689119:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203042.689123:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203042.689125:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203042.689129:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2314 00000100:00080000:3.0:1788203042.689132:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203042.689136:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203042.689147:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203042.689176:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442932864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203042.689179:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442932736:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203042.689185:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203042.689188:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203042.689190:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203042.689193:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2314 00000100:00080000:3.0:1788203042.689197:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203042.689201:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203042.689205:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203042.689214:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442932992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203042.689222:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203042.689225:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203042.689226:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203042.689229:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2314 00000100:00080000:3.0:1788203042.689232:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203042.689236:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203042.689239:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203042.689248:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442933120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203042.689258:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203042.689306:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203042/real 1788203042] req@ffff902352689500 x1875064442932736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203042.689326:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442932736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203042.689336:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203042.689340:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203042.689344:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442932736:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203042.689373:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442932864:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203042.689397:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203042.689411:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203042/real 1788203042] req@ffff902352688e00 x1875064442932864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203042.689422:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442932864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203042.689430:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203042.689433:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203042.689436:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442932864:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203042.689442:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442932992:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203042.689461:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203042.689472:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203042/real 1788203042] req@ffff90235268a680 x1875064442932992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203042.689489:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442932992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203042.689497:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203042.689500:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203042.689503:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442932992:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203042.689509:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442933120:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203042.689540:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203042.689551:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203042/real 1788203042] req@ffff902352689c00 x1875064442933120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203042.689575:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442933120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203042.689593:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203042.689596:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203042.689598:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442933120:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203043.522213:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203044.545040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203045.569035:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203046.593070:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203047.617040:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203047.808982:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203047.809001:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203047.809005:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203047.809018:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2319 00000100:00080000:1.0:1788203047.809031:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203047.809038:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203047.809076:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442933248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203047.809100:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203047.809103:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203047.809104:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203047.809107:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2319 00000100:00080000:1.0:1788203047.809110:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203047.809114:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203047.809125:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203047.809140:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442933376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203047.809151:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203047.809154:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203047.809156:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203047.809160:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2319 00000100:00080000:1.0:1788203047.809163:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203047.809166:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203047.809170:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203047.809177:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442933504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203047.809185:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203047.809191:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203047.809192:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788203047.809194:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442933248:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203047.809197:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2319 00000100:00080000:1.0:1788203047.809202:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203047.809209:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203047.809213:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203047.809230:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442933632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203047.809279:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203047.809331:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203047/real 1788203047] req@ffff90234519ea00 x1875064442933248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203047.809353:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442933248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203047.809366:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203047.809371:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203047.809375:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442933248:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203047.809383:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442933376:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203047.809407:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203047.809420:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203047/real 1788203047] req@ffff90234519c700 x1875064442933376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203047.809433:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442933376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203047.809451:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203047.809456:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203047.809459:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442933376:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203047.809498:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442933504:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203047.809520:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203047.809531:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203047/real 1788203047] req@ffff90234519fb80 x1875064442933504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203047.809543:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442933504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203047.809552:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203047.809556:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203047.809560:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442933504:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203047.809566:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442933632:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203047.809593:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203047.809604:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203047/real 1788203047] req@ffff90234519df80 x1875064442933632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203047.809616:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442933632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203047.809624:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203047.809627:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203047.809631:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442933632:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203048.641007:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203049.665058:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203050.689147:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203051.713043:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203052.737029:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203052.928958:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203052.928966:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203052.928970:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203052.928977:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2324 00000100:00080000:1.0:1788203052.928985:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203052.928991:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203052.929024:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442933760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203052.929051:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203052.929056:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203052.929058:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203052.929062:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2324 00000100:00080000:1.0:1788203052.929066:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203052.929070:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203052.929078:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203052.929091:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442933888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203052.929102:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203052.929105:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203052.929107:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203052.929118:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2324 00000100:00080000:1.0:1788203052.929122:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788203052.929127:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442933760:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203052.929128:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203052.929137:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203052.929148:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442934016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203052.929156:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203052.929158:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203052.929160:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203052.929162:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2324 00000100:00080000:1.0:1788203052.929165:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203052.929168:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203052.929172:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203052.929179:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442934144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203052.929192:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203052.929228:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203052/real 1788203052] req@ffff90234519ed80 x1875064442933760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203052.929244:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442933760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203052.929253:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203052.929256:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203052.929260:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442933760:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203052.929264:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442933888:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203052.929279:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203052.929288:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203052/real 1788203052] req@ffff90234519c000 x1875064442933888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203052.929297:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442933888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203052.929302:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203052.929305:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203052.929308:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442933888:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203052.929324:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442934016:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203052.929333:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203052.929339:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203052/real 1788203052] req@ffff90234519e300 x1875064442934016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203052.929346:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442934016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203052.929350:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203052.929351:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203052.929353:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442934016:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203052.929356:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442934144:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203052.929364:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203052.929369:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203052/real 1788203052] req@ffff90234519f800 x1875064442934144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203052.929375:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442934144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203052.929401:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203052.929404:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203052.929406:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442934144:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203053.761040:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203054.785075:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203055.808976:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203056.833007:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203057.857063:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203058.048989:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203058.049001:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203058.049005:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203058.049017:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2329 00000100:00080000:1.0:1788203058.049028:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203058.049035:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203058.049073:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442934272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203058.049107:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203058.049111:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203058.049113:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203058.049118:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2329 00000100:00080000:1.0:1788203058.049121:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203058.049126:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203058.049136:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203058.049153:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442934400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203058.049155:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442934272:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203058.049173:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203058.049177:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203058.049179:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203058.049185:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2329 00000100:00080000:1.0:1788203058.049190:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203058.049196:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203058.049202:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203058.049217:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442934528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203058.049228:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788203058.049229:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203058.049235:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203058.049238:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203058.049244:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2329 00000100:00080000:1.0:1788203058.049248:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203058.049253:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203058.049259:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203058.049272:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203058/real 1788203058] req@ffff90234519d880 x1875064442934272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203058.049275:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442934656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203058.049324:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442934272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203058.049333:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203058.049336:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203058.049339:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442934272:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203058.049358:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442934400:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203058.049375:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203058.049384:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203058/real 1788203058] req@ffff90234519f800 x1875064442934400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203058.049392:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442934400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203058.049399:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203058.049401:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203058.049403:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442934400:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203058.049408:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442934528:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203058.049419:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203058.049425:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203058/real 1788203058] req@ffff90234519e300 x1875064442934528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203058.049432:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442934528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203058.049439:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203058.049441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203058.049445:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442934528:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203058.049451:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442934656:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203058.049469:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203058.049480:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203058/real 1788203058] req@ffff90234519c000 x1875064442934656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203058.049493:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442934656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203058.049501:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203058.049504:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203058.049506:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442934656:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203058.881035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203059.905117:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203060.929054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203061.953059:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203062.977034:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203063.168927:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203063.168938:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203063.168943:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203063.168953:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2334 00000100:00080000:1.0:1788203063.168963:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203063.168970:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203063.169006:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442934784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203063.169032:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203063.169036:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203063.169038:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203063.169042:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2334 00000100:00080000:1.0:1788203063.169046:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203063.169051:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203063.169061:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203063.169076:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442934784:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203063.169077:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442934912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203063.169094:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203063.169098:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203063.169100:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203063.169105:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2334 00000100:00080000:1.0:1788203063.169109:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203063.169114:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203063.169119:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203063.169132:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442935040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203063.169144:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788203063.169144:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203063.169149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203063.169152:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203063.169158:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2334 00000100:00080000:1.0:1788203063.169163:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203063.169167:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203063.169172:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203063.169187:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442935168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203063.169188:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203063/real 1788203063] req@ffff90234519ed80 x1875064442934784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203063.169232:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442934784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203063.169241:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203063.169245:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203063.169248:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442934784:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203063.169266:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442934912:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203063.169282:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203063.169291:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203063/real 1788203063] req@ffff90234519df80 x1875064442934912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203063.169299:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442934912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203063.169305:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203063.169308:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203063.169311:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442934912:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203063.169317:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442935040:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203063.169336:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203063.169348:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203063/real 1788203063] req@ffff90234519fb80 x1875064442935040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203063.169361:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442935040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203063.169373:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203063.169375:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203063.169377:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442935040:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203063.169382:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442935168:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203063.169402:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203063.169413:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203063/real 1788203063] req@ffff90234519c700 x1875064442935168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203063.169426:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442935168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203063.169435:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203063.169438:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203063.169440:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442935168:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203064.001024:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203065.025014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203066.049043:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203067.073082:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203068.097092:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203068.288975:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203068.288983:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203068.288986:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203068.288995:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2339 00000100:00080000:1.0:1788203068.289004:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203068.289010:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203068.289042:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442935296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203068.289068:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203068.289071:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203068.289072:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203068.289075:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2339 00000100:00080000:1.0:1788203068.289078:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203068.289080:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203068.289088:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203068.289099:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442935424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203068.289106:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203068.289109:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203068.289111:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203068.289130:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2339 00000100:00080000:1.0:1788203068.289134:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203068.289137:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203068.289140:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203068.289150:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442935296:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203068.289151:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442935552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203068.289165:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203068.289167:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203068.289168:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203068.289171:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2339 00000100:00080000:1.0:1788203068.289173:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203068.289175:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203068.289178:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203068.289189:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442935680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203068.289212:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203068.289253:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203068/real 1788203068] req@ffff90234519ea00 x1875064442935296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203068.289267:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442935296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203068.289276:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203068.289281:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203068.289286:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442935296:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203068.289294:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442935424:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203068.289318:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203068.289331:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203068/real 1788203068] req@ffff90234519c380 x1875064442935424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203068.289342:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442935424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203068.289348:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203068.289350:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203068.289352:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442935424:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203068.289378:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442935552:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203068.289393:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203068.289402:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203068/real 1788203068] req@ffff90234519f480 x1875064442935552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203068.289414:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442935552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203068.289424:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203068.289427:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203068.289431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442935552:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203068.289438:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442935680:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203068.289458:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203068.289468:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203068/real 1788203068] req@ffff90234519dc00 x1875064442935680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203068.289477:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442935680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203068.289482:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203068.289484:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203068.289486:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442935680:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203069.121091:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203070.145109:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203071.169120:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203072.193023:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203073.217121:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203073.409076:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203073.409086:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203073.409091:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203073.409102:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2344 00000100:00080000:3.0:1788203073.409113:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203073.409120:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203073.409158:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442935808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203073.409196:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203073.409200:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203073.409202:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203073.409207:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2344 00000100:00080000:3.0:1788203073.409211:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203073.409215:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203073.409225:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203073.409239:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442935936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203073.409248:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442935808:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203073.409258:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203073.409266:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203073.409268:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203073.409273:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2344 00000100:00080000:3.0:1788203073.409278:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203073.409283:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203073.409290:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203073.409304:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442936064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203073.409308:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788203073.409321:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203073.409326:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203073.409328:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203073.409333:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2344 00000100:00080000:3.0:1788203073.409338:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203073.409342:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203073.409349:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203073.409355:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203073/real 1788203073] req@ffff902352689180 x1875064442935808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203073.409361:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442936192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203073.409406:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442935808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203073.409416:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203073.409419:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203073.409422:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442935808:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203073.409442:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442935936:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203073.409462:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203073.409471:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203073/real 1788203073] req@ffff902352688700 x1875064442935936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203073.409479:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442935936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203073.409489:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203073.409491:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203073.409494:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442935936:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203073.409497:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442936064:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203073.409509:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203073.409515:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203073/real 1788203073] req@ffff90235268bb80 x1875064442936064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203073.409522:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442936064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203073.409527:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203073.409529:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203073.409531:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442936064:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203073.409535:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442936192:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203073.409544:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203073.409550:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203073/real 1788203073] req@ffff902352689880 x1875064442936192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203073.409557:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442936192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203073.409561:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203073.409563:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203073.409566:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442936192:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203074.241051:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203075.265134:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203076.289087:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203077.313070:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203078.337099:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203078.528996:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203078.529010:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203078.529016:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203078.529028:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2350 00000100:00080000:3.0:1788203078.529042:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203078.529050:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203078.529100:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442936320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203078.529134:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203078.529139:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203078.529141:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203078.529146:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2350 00000100:00080000:3.0:1788203078.529150:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203078.529155:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203078.529168:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203078.529184:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442936448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203078.529197:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442936320:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203078.529197:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203078.529204:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203078.529208:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203078.529213:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2350 00000100:00080000:3.0:1788203078.529217:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203078.529221:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203078.529226:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203078.529240:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442936576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203078.529249:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203078.529252:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203078.529253:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203078.529256:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2350 00000100:00080000:3.0:1788203078.529260:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203078.529263:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203078.529268:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203078.529277:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442936704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203078.529285:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203078.529331:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203078/real 1788203078] req@ffff902352688380 x1875064442936320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203078.529347:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442936320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203078.529356:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203078.529359:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203078.529363:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442936320:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203078.529387:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442936448:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203078.529403:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203078.529412:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203078/real 1788203078] req@ffff90235268ad80 x1875064442936448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203078.529430:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442936448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203078.529437:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203078.529439:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203078.529442:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442936448:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203078.529446:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442936576:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203078.529459:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203078.529467:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203078/real 1788203078] req@ffff90235268b480 x1875064442936576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203078.529475:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442936576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203078.529481:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203078.529482:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203078.529485:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442936576:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203078.529488:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442936704:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203078.529499:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203078.529505:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203078/real 1788203078] req@ffff902352688a80 x1875064442936704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203078.529512:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442936704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203078.529518:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203078.529520:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203078.529523:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442936704:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203079.361084:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203080.385065:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203081.409070:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203082.433046:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203083.457078:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203083.648951:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203083.648959:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203083.648962:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203083.648970:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2355 00000100:00080000:3.0:1788203083.648979:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203083.648985:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203083.649022:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064442936832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203083.649053:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203083.649056:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203083.649058:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203083.649061:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2355 00000100:00080000:3.0:1788203083.649064:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203083.649067:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203083.649074:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203083.649090:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442936960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203083.649102:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442936832:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203083.649115:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203083.649122:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203083.649124:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203083.649130:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2355 00000100:00080000:3.0:1788203083.649133:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203083.649136:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203083.649141:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203083.649158:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442937088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203083.649168:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788203083.649168:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203083.649174:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203083.649175:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203083.649179:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2355 00000100:00080000:3.0:1788203083.649183:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203083.649186:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203083.649189:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203083.649198:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442937216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203083.649222:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203083/real 1788203083] req@ffff90235268aa00 x1875064442936832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203083.649252:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064442936832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203083.649262:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203083.649265:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203083.649269:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442936832:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203083.649290:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442936960:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203083.649315:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203083.649329:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203083/real 1788203083] req@ffff902352688a80 x1875064442936960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203083.649350:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442936960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203083.649357:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203083.649359:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203083.649361:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442936960:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203083.649366:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442937088:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203083.649383:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203083.649391:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203083/real 1788203083] req@ffff90235268b480 x1875064442937088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203083.649399:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442937088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203083.649404:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203083.649412:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203083.649415:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442937088:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203083.649419:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442937216:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203083.649431:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203083.649438:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203083/real 1788203083] req@ffff90235268ad80 x1875064442937216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203083.649445:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442937216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203083.649450:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203083.649452:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203083.649454:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442937216:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203084.481036:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203085.505051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203086.529076:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203087.553091:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203088.577082:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203088.769020:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203088.769036:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203088.769041:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203088.769055:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2360 00000100:00080000:3.0:1788203088.769070:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203088.769080:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203088.769133:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442937344/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203088.769171:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203088.769177:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203088.769179:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203088.769185:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2360 00000100:00080000:3.0:1788203088.769190:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203088.769196:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203088.769209:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203088.769227:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442937472/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203088.769233:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442937344:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203088.769246:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203088.769251:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203088.769253:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203088.769258:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2360 00000100:00080000:3.0:1788203088.769263:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203088.769268:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203088.769275:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203088.769290:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442937600/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203088.769302:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203088.769307:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203088.769309:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203088.769313:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2360 00000100:00080000:3.0:1788203088.769319:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788203088.769320:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788203088.769336:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203088.769343:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203088.769357:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442937728/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203088.769385:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203088/real 1788203088] req@ffff902352688380 x1875064442937344/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203088.769410:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442937344/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203088.769421:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203088.769424:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203088.769428:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442937344:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203088.769452:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442937472:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203088.769472:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203088.769485:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203088/real 1788203088] req@ffff902352689880 x1875064442937472/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203088.769495:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442937472/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203088.769500:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203088.769502:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203088.769505:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442937472:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203088.769511:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442937600:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203088.769534:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203088.769541:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203088/real 1788203088] req@ffff90235268bb80 x1875064442937600/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203088.769549:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442937600/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203088.769555:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203088.769558:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203088.769560:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442937600:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203088.769564:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442937728:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203088.769577:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203088.769585:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203088/real 1788203088] req@ffff902352688700 x1875064442937728/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203088.769593:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442937728/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203088.769599:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203088.769602:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203088.769604:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442937728:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203089.601021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203090.625074:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203091.649020:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203092.673081:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203093.697053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203093.889049:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203093.889060:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203093.889065:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203093.889077:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2365 00000100:00080000:1.0:1788203093.889089:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203093.889098:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203093.889144:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442937856/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203093.889173:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203093.889177:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203093.889178:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203093.889181:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2365 00000100:00080000:1.0:1788203093.889184:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203093.889186:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203093.889197:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203093.889213:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442937984/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203093.889226:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203093.889230:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203093.889233:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203093.889238:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2365 00000100:00080000:1.0:1788203093.889243:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203093.889249:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203093.889255:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203093.889261:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442937856:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203093.889270:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442938112/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203093.889279:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203093.889283:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203093.889285:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203093.889291:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2365 00000100:00080000:1.0:1788203093.889295:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203093.889298:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203093.889302:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203093.889313:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442938240/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203093.889335:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203093.889394:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203093/real 1788203093] req@ffff90234519f100 x1875064442937856/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203093.889415:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442937856/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203093.889424:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203093.889428:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203093.889431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442937856:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203093.889438:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442937984:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203093.889457:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203093.889476:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203093/real 1788203093] req@ffff90234519dc00 x1875064442937984/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203093.889484:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442937984/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203093.889490:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203093.889493:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203093.889495:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442937984:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203093.889515:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442938112:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203093.889529:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203093.889536:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203093/real 1788203093] req@ffff90234519f480 x1875064442938112/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203093.889551:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442938112/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203093.889556:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203093.889559:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203093.889561:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442938112:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203093.889565:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442938240:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203093.889578:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203093.889584:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203093/real 1788203093] req@ffff90234519c380 x1875064442938240/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203093.889592:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442938240/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203093.889596:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203093.889599:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203093.889600:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442938240:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203094.721016:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203095.745019:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203096.769103:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203097.793079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203098.817037:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203099.009014:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203099.009028:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203099.009033:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203099.009045:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2370 00000100:00080000:1.0:1788203099.009058:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203099.009067:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203099.009121:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442938368/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203099.009155:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203099.009160:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203099.009162:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203099.009167:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2370 00000100:00080000:1.0:1788203099.009171:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203099.009176:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203099.009189:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203099.009209:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442938496/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203099.009223:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442938368:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203099.009229:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203099.009238:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203099.009241:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203099.009246:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2370 00000100:00080000:1.0:1788203099.009250:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203099.009254:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203099.009259:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203099.009269:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442938624/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203099.009276:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203099.009279:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203099.009281:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203099.009283:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2370 00000100:00080000:1.0:1788203099.009286:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203099.009291:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203099.009294:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203099.009301:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442938752/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203099.009325:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203099.009378:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203099/real 1788203099] req@ffff90234519ea00 x1875064442938368/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203099.009399:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442938368/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203099.009411:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203099.009416:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203099.009420:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442938368:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203099.009446:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442938496:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203099.009466:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203099.009476:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203099/real 1788203099] req@ffff90234519c700 x1875064442938496/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203099.009485:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442938496/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203099.009491:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203099.009494:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203099.009496:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442938496:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203099.009501:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442938624:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203099.009512:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203099.009518:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203099/real 1788203099] req@ffff90234519fb80 x1875064442938624/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203099.009531:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442938624/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203099.009536:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203099.009538:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203099.009540:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442938624:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203099.009544:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442938752:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203099.009554:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203099.009560:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203099/real 1788203099] req@ffff90234519df80 x1875064442938752/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203099.009567:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442938752/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203099.009572:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203099.009574:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203099.009576:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442938752:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203099.841180:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203100.865080:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203101.889076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203102.913031:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203103.937030:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203104.129018:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203104.129026:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203104.129030:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203104.129038:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2375 00000100:00080000:0.0:1788203104.129052:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203104.129058:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203104.129097:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442938880/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203104.129130:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203104.129139:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203104.129141:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203104.129145:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2375 00000100:00080000:0.0:1788203104.129150:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203104.129156:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203104.129166:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203104.129176:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442938880:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203104.129184:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442939008/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203104.129199:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203104.129202:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203104.129205:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203104.129209:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2375 00000100:00080000:0.0:1788203104.129215:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203104.129220:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203104.129236:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203104.129250:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442939136/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203104.129258:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203104.129261:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203104.129263:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203104.129266:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2375 00000100:00080000:0.0:1788203104.129271:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203104.129277:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203104.129278:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442939008:192.168.201.130@tcp:38:kworker.0 00000008:00100000:0.0:1788203104.129284:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203104.129296:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442939264/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203104.129311:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442939136:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203104.129331:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442939264:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203104.961230:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203105.985399:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203107.009099:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788203107.201154:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:1.0:1788203107.201188:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788203107.201202:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788203107.201235:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788203107.201241:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788203107.201247:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:2.0:1788203107.201285:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203104/real 1788203107] req@ffff9023450bad80 x1875064442939264/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203107.201318:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442939264/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203107.201327:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203107.201331:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203107.201335:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442939264:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203107.201341:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203104/real 1788203107] req@ffff9023450b9c00 x1875064442939008/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203107.201354:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442939008/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203107.201364:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203107.201368:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203107.201371:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442939008:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203107.201378:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203104/real 1788203107] req@ffff9023450b9f80 x1875064442939136/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203107.201389:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442939136/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203107.201397:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203107.201401:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203107.201404:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442939136:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203107.201411:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203104/real 1788203107] req@ffff9023450b8700 x1875064442938880/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203107.201422:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442938880/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203107.201429:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203107.201432:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203107.201452:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442938880:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788203108.033010:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203109.056998:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203109.249026:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203109.249036:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203109.249040:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203109.249050:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2380 00000100:00080000:1.0:1788203109.249061:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203109.249067:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203109.249099:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442939392/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203109.249131:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203109.249135:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203109.249137:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203109.249141:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2380 00000100:00080000:1.0:1788203109.249145:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203109.249148:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203109.249158:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203109.249170:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442939520/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203109.249181:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203109.249184:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203109.249186:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203109.249190:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2380 00000100:00080000:1.0:1788203109.249193:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203109.249196:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203109.249201:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203109.249210:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442939648/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203109.249219:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203109.249222:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203109.249223:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203109.249227:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2380 00000100:00080000:1.0:1788203109.249231:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203109.249233:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203109.249239:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203109.249248:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442939776/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203109.249463:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442939392:192.168.201.130@tcp:250:kworker.0 00000800:00000100:2.0:1788203109.249549:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203109.249600:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203109/real 1788203109] req@ffff90234519ed80 x1875064442939392/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203109.249621:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442939392/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203109.249634:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203109.249638:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203109.249642:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442939392:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203109.249650:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442939520:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203109.249676:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203109.249690:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203109/real 1788203109] req@ffff90234519c000 x1875064442939520/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203109.249702:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442939520/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203109.249712:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203109.249715:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203109.249719:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442939520:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203109.249726:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442939648:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203109.249748:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203109.249761:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203109/real 1788203109] req@ffff90234519e300 x1875064442939648/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203109.249773:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442939648/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203109.249782:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203109.249785:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203109.249789:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442939648:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203109.249795:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442939776:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203109.249817:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203109.249830:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203109/real 1788203109] req@ffff90234519f800 x1875064442939776/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203109.249843:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442939776/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203109.249851:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203109.249854:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203109.249859:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442939776:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203110.081032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203111.105021:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203112.129059:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203113.153053:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203114.177083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203114.369036:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203114.369048:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203114.369052:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203114.369064:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2385 00000100:00080000:3.0:1788203114.369077:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203114.369084:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203114.369131:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442939904/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203114.369157:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203114.369161:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203114.369162:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203114.369165:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2385 00000100:00080000:3.0:1788203114.369168:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203114.369170:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203114.369180:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203114.369197:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442940032/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203114.369204:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203114.369209:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203114.369211:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203114.369214:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2385 00000100:00080000:3.0:1788203114.369216:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203114.369222:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442939904:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203114.369223:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203114.369228:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203114.369239:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442940160/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203114.369247:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203114.369250:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203114.369251:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203114.369253:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2385 00000100:00080000:3.0:1788203114.369255:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203114.369258:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203114.369261:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203114.369269:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442940288/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203114.369347:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203114.369402:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203114/real 1788203114] req@ffff902352689180 x1875064442939904/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203114.369424:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442939904/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203114.369436:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203114.369440:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203114.369445:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442939904:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203114.369453:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442940032:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203114.369477:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203114.369490:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203114/real 1788203114] req@ffff902352689c00 x1875064442940032/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203114.369503:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442940032/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203114.369511:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203114.369515:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203114.369518:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442940032:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203114.369551:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442940160:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203114.369571:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203114.369582:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203114/real 1788203114] req@ffff90235268a680 x1875064442940160/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203114.369594:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442940160/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203114.369603:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203114.369607:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203114.369611:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442940160:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203114.369617:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442940288:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203114.369634:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203114.369644:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203114/real 1788203114] req@ffff902352688e00 x1875064442940288/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203114.369656:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442940288/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203114.369664:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203114.369667:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203114.369671:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442940288:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203115.201064:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203116.225050:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203117.249028:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203118.273087:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203119.297065:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203119.489004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203119.489018:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203119.489023:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203119.489036:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2391 00000100:00080000:0.0:1788203119.489051:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203119.489059:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203119.489112:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442940416/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203119.489148:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203119.489153:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203119.489155:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203119.489159:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2391 00000100:00080000:0.0:1788203119.489163:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203119.489169:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203119.489182:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203119.489198:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442940544/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203119.489208:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442940416:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203119.489214:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203119.489241:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203119.489243:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203119.489257:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2391 00000100:00080000:0.0:1788203119.489263:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203119.489268:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203119.489276:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203119.489292:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442940672/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203119.489304:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203119.489309:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203119.489311:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203119.489315:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2391 00000100:00080000:0.0:1788203119.489319:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788203119.489322:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788203119.489330:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203119.489337:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203119.489352:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442940800/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203119.489387:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203119/real 1788203119] req@ffff9023450baa00 x1875064442940416/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203119.489409:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442940416/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203119.489422:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203119.489426:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203119.489431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442940416:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203119.489458:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442940544:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203119.489484:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203119.489497:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203119/real 1788203119] req@ffff9023450b9180 x1875064442940544/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203119.489511:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442940544/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203119.489520:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203119.489524:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203119.489527:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442940544:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203119.489534:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442940672:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203119.489555:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203119.489567:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203119/real 1788203119] req@ffff9023450b8000 x1875064442940672/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203119.489587:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442940672/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203119.489595:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203119.489599:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203119.489602:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442940672:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203119.489609:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442940800:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203119.489628:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203119.489639:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203119/real 1788203119] req@ffff9023450ba300 x1875064442940800/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203119.489651:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442940800/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203119.489659:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203119.489663:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203119.489666:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442940800:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203120.321037:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203121.345054:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203122.369094:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203123.393025:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203124.417100:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203124.608989:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203124.608999:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203124.609003:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203124.609012:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2396 00000100:00080000:2.0:1788203124.609022:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203124.609029:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203124.609068:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442940928/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203124.609098:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203124.609102:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203124.609103:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203124.609119:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2396 00000100:00080000:2.0:1788203124.609125:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203124.609131:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203124.609155:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203124.609174:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442941056/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203124.609188:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203124.609191:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203124.609193:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203124.609199:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2396 00000100:00080000:2.0:1788203124.609204:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203124.609209:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203124.609214:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203124.609224:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442941184/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203124.609233:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203124.609236:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203124.609239:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203124.609243:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2396 00000100:00080000:2.0:1788203124.609250:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203124.609255:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203124.609259:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203124.609271:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442941312/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203124.609296:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442940928:192.168.201.130@tcp:250:kworker.0 00000800:00000100:3.0:1788203124.609456:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203124.609496:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203124/real 1788203124] req@ffff90234598c380 x1875064442940928/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203124.609513:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442940928/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203124.609528:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203124.609531:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203124.609535:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442940928:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203124.609542:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442941056:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203124.609701:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203124.609729:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203124/real 1788203124] req@ffff90234598d180 x1875064442941056/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203124.609746:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442941056/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203124.609808:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203124.609815:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203124.609822:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442941056:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203124.609832:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442941184:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203124.609854:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203124.609864:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203124/real 1788203124] req@ffff90234598e300 x1875064442941184/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203124.609873:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442941184/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203124.609914:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203124.609917:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203124.609923:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442941184:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203124.609952:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442941312:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203124.609967:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203124.609976:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203124/real 1788203124] req@ffff90234598df80 x1875064442941312/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203124.609984:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442941312/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203124.609990:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203124.609993:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203124.609995:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442941312:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203125.441015:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203126.465060:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203127.489129:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203128.513045:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203129.537134:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203129.729032:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203129.729043:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203129.729067:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203129.729078:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2401 00000100:00080000:3.0:1788203129.729091:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203129.729097:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203129.729144:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442941440/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203129.729179:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203129.729184:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203129.729186:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203129.729191:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2401 00000100:00080000:3.0:1788203129.729196:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203129.729201:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203129.729212:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203129.729221:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442941440:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203129.729227:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442941568/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203129.729237:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203129.729239:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203129.729241:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203129.729244:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2401 00000100:00080000:3.0:1788203129.729246:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203129.729249:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203129.729252:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203129.729260:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442941696/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203129.729267:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203129.729270:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203129.729271:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203129.729273:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2401 00000100:00080000:3.0:1788203129.729298:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203129.729301:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203129.729304:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203129.729314:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442941824/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203129.729316:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203129.729388:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203129/real 1788203129] req@ffff902352689500 x1875064442941440/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203129.729413:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442941440/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203129.729426:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203129.729431:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203129.729436:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442941440:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203129.729467:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442941568:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203129.729495:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203129.729511:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203129/real 1788203129] req@ffff90235268b800 x1875064442941568/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203129.729524:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442941568/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203129.729534:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203129.729538:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203129.729541:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442941568:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203129.729548:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442941696:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203129.729568:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203129.729580:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203129/real 1788203129] req@ffff90235268a300 x1875064442941696/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203129.729592:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442941696/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203129.729602:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203129.729606:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203129.729609:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442941696:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203129.729625:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442941824:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203129.729642:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203129.729651:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203129/real 1788203129] req@ffff90235268b100 x1875064442941824/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203129.729659:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442941824/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203129.729665:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203129.729674:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203129.729677:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442941824:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203130.561125:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203131.585096:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203132.609039:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203133.633120:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203134.657066:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203134.848978:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203134.848990:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203134.848994:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203134.849005:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2406 00000100:00080000:3.0:1788203134.849017:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203134.849025:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203134.849075:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442941952/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203134.849100:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203134.849102:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203134.849104:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203134.849108:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2406 00000100:00080000:3.0:1788203134.849110:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203134.849113:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203134.849123:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203134.849135:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442942080/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203134.849144:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788203134.849146:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442941952:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203134.849149:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203134.849159:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203134.849166:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2406 00000100:00080000:3.0:1788203134.849168:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203134.849171:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203134.849175:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203134.849185:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442942208/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203134.849197:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203134.849201:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203134.849203:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203134.849207:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2406 00000100:00080000:3.0:1788203134.849222:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203134.849227:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203134.849233:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788203134.849235:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203134.849251:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442942336/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203134.849286:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203134/real 1788203134] req@ffff902352688000 x1875064442941952/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203134.849304:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442941952/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203134.849313:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203134.849325:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203134.849331:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442941952:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203134.849355:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442942080:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203134.849373:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203134.849381:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203134/real 1788203134] req@ffff90235268b100 x1875064442942080/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203134.849390:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442942080/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203134.849396:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203134.849399:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203134.849401:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442942080:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203134.849407:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442942208:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203134.849419:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203134.849425:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203134/real 1788203134] req@ffff90235268a300 x1875064442942208/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203134.849433:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442942208/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203134.849438:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203134.849440:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203134.849442:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442942208:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203134.849446:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442942336:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203134.849456:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203134.849461:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203134/real 1788203134] req@ffff90235268b800 x1875064442942336/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203134.849468:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442942336/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203134.849473:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203134.849475:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203134.849477:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442942336:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203135.681142:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203136.705110:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203137.729091:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203138.753149:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203139.777076:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203139.969001:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203139.969014:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203139.969018:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203139.969029:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2411 00000100:00080000:2.0:1788203139.969042:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203139.969056:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203139.969102:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442942464/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203139.969131:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203139.969135:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203139.969136:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203139.969140:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2411 00000100:00080000:2.0:1788203139.969143:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203139.969145:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203139.969154:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203139.969164:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442942592/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203139.969171:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203139.969174:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203139.969176:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203139.969178:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2411 00000100:00080000:2.0:1788203139.969180:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203139.969182:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203139.969185:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203139.969188:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442942464:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203139.969195:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442942720/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203139.969204:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203139.969207:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203139.969208:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203139.969211:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2411 00000100:00080000:2.0:1788203139.969213:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203139.969215:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203139.969224:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203139.969234:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442942848/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203139.969267:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203139.969313:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203139/real 1788203139] req@ffff90234598f100 x1875064442942464/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203139.969331:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442942464/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203139.969341:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203139.969344:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203139.969347:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442942464:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203139.969353:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442942592:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203139.969369:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203139.969378:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203139/real 1788203139] req@ffff90234598ed80 x1875064442942592/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203139.969387:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442942592/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203139.969393:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203139.969395:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203139.969397:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442942592:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203139.969420:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442942720:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203139.969431:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203139.969441:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203139/real 1788203139] req@ffff90234598fb80 x1875064442942720/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203139.969454:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442942720/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203139.969463:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203139.969466:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203139.969470:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442942720:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203139.969476:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442942848:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203139.969497:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203139.969508:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203139/real 1788203139] req@ffff90234598df80 x1875064442942848/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203139.969528:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442942848/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203139.969539:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203139.969541:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203139.969543:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442942848:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203140.801092:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203141.825090:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203142.849037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203143.873085:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203144.897074:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203145.089062:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203145.089077:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203145.089083:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203145.089097:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2416 00000100:00080000:0.0:1788203145.089112:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203145.089120:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203145.089174:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442942976/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203145.089211:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203145.089216:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203145.089218:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203145.089223:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2416 00000100:00080000:0.0:1788203145.089227:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203145.089232:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203145.089244:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203145.089260:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442943104/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203145.089272:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203145.089278:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203145.089280:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203145.089283:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442942976:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203145.089287:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2416 00000100:00080000:0.0:1788203145.089293:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203145.089298:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203145.089304:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203145.089318:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442943232/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203145.089329:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203145.089334:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203145.089336:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203145.089340:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2416 00000100:00080000:0.0:1788203145.089345:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203145.089350:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203145.089355:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203145.089365:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203145.089370:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442943360/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203145.089416:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203145/real 1788203145] req@ffff9023450ba300 x1875064442942976/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203145.089434:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442942976/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203145.089443:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203145.089447:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203145.089450:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442942976:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203145.089474:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442943104:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203145.089490:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203145.089499:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203145/real 1788203145] req@ffff9023450b8000 x1875064442943104/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203145.089507:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442943104/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203145.089513:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203145.089515:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203145.089518:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442943104:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203145.089522:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442943232:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203145.089535:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203145.089543:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203145/real 1788203145] req@ffff9023450b9180 x1875064442943232/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203145.089556:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442943232/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203145.089565:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203145.089569:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203145.089573:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442943232:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203145.089579:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442943360:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203145.089601:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203145.089611:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203145/real 1788203145] req@ffff9023450baa00 x1875064442943360/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203145.089619:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442943360/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203145.089624:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203145.089627:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203145.089629:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442943360:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203145.921095:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203146.945078:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203147.969028:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203148.993069:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203150.017041:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203150.209015:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203150.209025:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203150.209030:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203150.209041:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2421 00000100:00080000:0.0:1788203150.209053:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203150.209061:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203150.209104:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442943488/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203150.209143:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203150.209146:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203150.209148:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203150.209151:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2421 00000100:00080000:0.0:1788203150.209155:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203150.209160:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203150.209171:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203150.209188:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442943616/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203150.209199:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442943488:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203150.209203:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203150.209211:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203150.209213:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203150.209218:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2421 00000100:00080000:0.0:1788203150.209221:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203150.209225:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203150.209229:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203150.209241:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442943744/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203150.209249:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203150.209252:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203150.209253:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203150.209255:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2421 00000100:00080000:0.0:1788203150.209258:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203150.209261:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203150.209264:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203150.209272:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442943872/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203150.209283:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203150.209345:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203150/real 1788203150] req@ffff9023450b8700 x1875064442943488/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203150.209362:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442943488/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203150.209371:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203150.209374:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203150.209377:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442943488:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203150.209403:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442943616:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203150.209421:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203150.209439:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203150/real 1788203150] req@ffff9023450b9f80 x1875064442943616/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203150.209448:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442943616/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203150.209454:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203150.209456:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203150.209458:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442943616:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203150.209461:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442943744:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203150.209474:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203150.209481:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203150/real 1788203150] req@ffff9023450b9c00 x1875064442943744/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203150.209488:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442943744/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203150.209498:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203150.209501:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203150.209503:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442943744:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203150.209506:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442943872:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203150.209516:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203150.209521:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203150/real 1788203150] req@ffff9023450bad80 x1875064442943872/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203150.209528:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442943872/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203150.209534:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203150.209535:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203150.209537:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442943872:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203151.041096:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203152.065083:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203153.089085:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203154.113094:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203155.137046:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203155.329004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203155.329016:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203155.329020:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203155.329035:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2426 00000100:00080000:1.0:1788203155.329058:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203155.329067:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203155.329117:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442944000/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203155.329146:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203155.329149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203155.329150:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203155.329155:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2426 00000100:00080000:1.0:1788203155.329158:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203155.329162:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203155.329175:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203155.329195:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442944128/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203155.329207:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442944000:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203155.329226:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203155.329231:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203155.329232:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203155.329237:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2426 00000100:00080000:1.0:1788203155.329243:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203155.329247:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203155.329253:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203155.329267:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442944256/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203155.329275:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203155.329278:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203155.329280:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203155.329283:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2426 00000100:00080000:1.0:1788203155.329286:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788203155.329292:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788203155.329297:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203155.329301:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203155.329312:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442944384/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203155.329352:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203155/real 1788203155] req@ffff90234519d880 x1875064442944000/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203155.329378:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442944000/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203155.329390:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203155.329395:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203155.329399:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442944000:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203155.329421:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442944128:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203155.329441:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203155.329451:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203155/real 1788203155] req@ffff90234519f800 x1875064442944128/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203155.329460:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442944128/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203155.329465:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203155.329468:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203155.329471:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442944128:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203155.329475:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442944256:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203155.329487:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203155.329493:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203155/real 1788203155] req@ffff90234519e300 x1875064442944256/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203155.329500:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442944256/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203155.329506:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203155.329508:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203155.329510:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442944256:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203155.329514:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442944384:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203155.329523:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203155.329529:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203155/real 1788203155] req@ffff90234519c000 x1875064442944384/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203155.329537:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442944384/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203155.329542:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203155.329544:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203155.329546:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442944384:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203156.161045:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203157.185056:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203158.209057:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203159.233030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203160.257038:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203160.448981:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203160.448992:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203160.448996:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203160.449007:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2431 00000100:00080000:1.0:1788203160.449019:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203160.449027:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203160.449066:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442944512/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203160.449099:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203160.449103:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203160.449105:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203160.449110:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2431 00000100:00080000:1.0:1788203160.449114:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203160.449119:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203160.449129:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203160.449143:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442944512:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203160.449158:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442944640/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203160.449172:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203160.449176:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203160.449179:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203160.449184:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2431 00000100:00080000:1.0:1788203160.449189:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203160.449194:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203160.449201:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788203160.449202:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203160.449219:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442944768/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203160.449229:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788203160.449242:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203160/real 1788203160] req@ffff90234519ed80 x1875064442944512/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203160.449244:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203160.449249:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203160.449259:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2431 00000100:00080000:1.0:1788203160.449265:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203160.449265:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442944512/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203160.449275:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00080000:2.0:1788203160.449279:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1788203160.449285:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1788203160.449285:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203160.449291:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442944512:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203160.449304:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442944896/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203160.449315:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442944640:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203160.449335:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203160.449348:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203160/real 1788203160] req@ffff90234519df80 x1875064442944640/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203160.449359:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442944640/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203160.449365:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203160.449367:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203160.449369:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442944640:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203160.449373:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442944768:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203160.449384:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203160.449390:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203160/real 1788203160] req@ffff90234519fb80 x1875064442944768/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203160.449397:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442944768/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203160.449403:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203160.449404:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203160.449406:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442944768:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203160.449414:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442944896:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203160.449425:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203160.449431:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203160/real 1788203160] req@ffff90234519c700 x1875064442944896/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203160.449438:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442944896/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203160.449443:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203160.449445:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203160.449447:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442944896:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203161.281047:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203162.305032:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203163.329067:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203164.353013:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203165.377125:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203165.569023:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203165.569034:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203165.569037:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203165.569048:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2437 00000100:00080000:2.0:1788203165.569058:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203165.569066:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203165.569103:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442945024/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203165.569138:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203165.569142:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203165.569144:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203165.569148:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2437 00000100:00080000:2.0:1788203165.569160:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203165.569166:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203165.569179:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203165.569192:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442945024:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203165.569194:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442945152/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203165.569211:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203165.569214:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203165.569215:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203165.569219:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2437 00000100:00080000:2.0:1788203165.569223:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203165.569227:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203165.569231:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203165.569244:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442945280/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203165.569253:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203165.569256:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203165.569258:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203165.569262:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2437 00000800:00000100:3.0:1788203165.569267:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788203165.569269:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203165.569275:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203165.569280:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203165.569291:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442945408/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203165.569306:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203165/real 1788203165] req@ffff90234598e300 x1875064442945024/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203165.569324:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442945024/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203165.569333:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203165.569336:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203165.569339:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442945024:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203165.569360:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442945152:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203165.569376:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203165.569384:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203165/real 1788203165] req@ffff90234598d180 x1875064442945152/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203165.569393:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442945152/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203165.569403:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203165.569405:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203165.569407:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442945152:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203165.569411:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442945280:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203165.569446:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203165.569456:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203165/real 1788203165] req@ffff90234598c380 x1875064442945280/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203165.569464:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442945280/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203165.569470:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203165.569472:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203165.569474:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442945280:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203165.569478:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442945408:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203165.569491:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203165.569497:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203165/real 1788203165] req@ffff90234598f480 x1875064442945408/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203165.569504:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442945408/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203165.569509:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203165.569510:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203165.569513:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442945408:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203166.401120:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203167.425035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203168.449126:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203169.473088:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203170.497090:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203170.689062:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203170.689073:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203170.689077:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203170.689089:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2442 00000100:00080000:0.0:1788203170.689101:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203170.689109:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203170.689156:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442945536/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203170.689195:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203170.689199:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203170.689201:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203170.689206:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2442 00000100:00080000:0.0:1788203170.689210:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203170.689215:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203170.689227:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203170.689239:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442945664/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203170.689246:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203170.689249:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203170.689250:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203170.689253:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2442 00000100:00080000:0.0:1788203170.689255:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203170.689258:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203170.689263:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203170.689276:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442945792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203170.689286:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203170.689290:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203170.689292:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203170.689296:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2442 00000100:00080000:0.0:1788203170.689301:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203170.689307:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203170.689312:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203170.689322:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442945920/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203170.689610:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442945536:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203170.689727:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442945664:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203170.689762:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442945792:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203170.689782:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442945920:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203171.521914:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203172.545038:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203173.569124:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788203173.761209:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788203173.761270:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788203173.761325:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788203173.761381:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788203173.761393:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788203173.761402:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788203173.761508:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203170/real 1788203173] req@ffff9023450b9500 x1875064442945536/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203173.761540:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442945536/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203173.761555:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203173.761561:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203173.761567:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442945536:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203173.761577:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203170/real 1788203173] req@ffff9023450bb800 x1875064442945664/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203173.761590:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442945664/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203173.761600:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203173.761604:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203173.761608:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442945664:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203173.761616:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203170/real 1788203173] req@ffff9023450b8a80 x1875064442945792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203173.761632:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442945792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203173.761641:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203173.761645:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203173.761649:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442945792:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203173.761656:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203170/real 1788203173] req@ffff9023450b8380 x1875064442945920/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203173.761669:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442945920/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203173.761677:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203173.761682:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203173.761685:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442945920:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203174.593281:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203175.617021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203175.809107:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203175.809118:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203175.809122:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203175.809133:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2447 00000100:00080000:2.0:1788203175.809144:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203175.809152:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203175.809189:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442946048/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203175.809220:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203175.809224:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203175.809226:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203175.809231:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2447 00000100:00080000:2.0:1788203175.809235:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203175.809240:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203175.809268:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203175.809285:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442946176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203175.809299:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788203175.809302:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442946048:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203175.809312:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203175.809315:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203175.809320:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2447 00000100:00080000:2.0:1788203175.809324:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203175.809330:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203175.809336:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203175.809352:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442946304/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203175.809364:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203175.809368:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203175.809370:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203175.809374:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2447 00000100:00080000:2.0:1788203175.809379:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788203175.809384:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788203175.809388:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203175.809395:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203175.809410:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442946432/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203175.809438:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203175/real 1788203175] req@ffff90234598dc00 x1875064442946048/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203175.809461:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442946048/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203175.809472:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203175.809477:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203175.809482:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442946048:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203175.809509:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442946176:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203175.809540:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203175.809558:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203175/real 1788203175] req@ffff90234598f800 x1875064442946176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203175.809572:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442946176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203175.809581:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203175.809584:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203175.809587:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442946176:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203175.809594:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442946304:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203175.809616:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203175.809628:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203175/real 1788203175] req@ffff90234598ce00 x1875064442946304/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203175.809642:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442946304/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203175.809650:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203175.809653:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203175.809657:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442946304:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203175.809664:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442946432:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203175.809686:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203175.809698:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203175/real 1788203175] req@ffff90234598c700 x1875064442946432/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203175.809710:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442946432/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203175.809717:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203175.809720:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203175.809723:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442946432:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203176.641179:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203177.665135:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203178.689077:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203179.713026:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203180.737108:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203180.928978:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203180.928986:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203180.928989:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203180.928997:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2452 00000100:00080000:2.0:1788203180.929007:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203180.929013:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203180.929048:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064442946560/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203180.929071:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203180.929074:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203180.929075:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203180.929078:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2452 00000100:00080000:2.0:1788203180.929080:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203180.929083:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203180.929090:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203180.929098:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064442946688/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203180.929106:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203180.929108:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203180.929109:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203180.929111:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2452 00000100:00080000:2.0:1788203180.929113:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203180.929117:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203180.929121:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203180.929128:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442946816/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203180.929135:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203180.929138:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203180.929140:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203180.929142:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2452 00000100:00080000:2.0:1788203180.929146:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203180.929167:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203180.929168:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442946560:192.168.201.130@tcp:250:kworker.0 00000008:00100000:2.0:1788203180.929172:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203180.929199:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442946944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203180.929256:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203180.929306:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203180/real 1788203180] req@ffff90234598ca80 x1875064442946560/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203180.929323:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064442946560/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203180.929331:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203180.929334:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203180.929337:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442946560:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203180.929343:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442946688:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203180.929358:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203180.929366:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203180/real 1788203180] req@ffff90234598e680 x1875064442946688/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203180.929373:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064442946688/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203180.929382:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203180.929384:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203180.929386:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442946688:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203180.929411:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442946816:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203180.929422:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203180.929427:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203180/real 1788203180] req@ffff90234598d880 x1875064442946816/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203180.929434:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442946816/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203180.929439:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203180.929441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203180.929442:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442946816:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203180.929446:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442946944:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203180.929455:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203180.929460:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203180/real 1788203180] req@ffff90234598c700 x1875064442946944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203180.929467:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442946944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203180.929471:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203180.929473:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203180.929475:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442946944:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203181.761012:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203182.785110:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203183.809134:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203184.833124:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203185.857110:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203186.049105:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203186.049117:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203186.049121:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203186.049132:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2457 00000100:00080000:1.0:1788203186.049144:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203186.049151:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203186.049203:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442947072/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203186.049246:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203186.049251:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203186.049253:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203186.049258:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2457 00000100:00080000:1.0:1788203186.049263:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203186.049267:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203186.049277:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203186.049299:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442947072:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203186.049305:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442947200/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203186.049320:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203186.049324:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203186.049326:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203186.049331:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2457 00000100:00080000:1.0:1788203186.049335:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203186.049349:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203186.049360:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203186.049372:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203186.049375:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442947328/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203186.049403:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203186.049407:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203186.049409:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203186.049414:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2457 00000100:00080000:1.0:1788203186.049419:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203186.049423:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203186.049428:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203186.049441:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203186/real 1788203186] req@ffff90234519ea00 x1875064442947072/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203186.049445:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442947456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203186.049482:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442947072/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203186.049495:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203186.049500:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203186.049505:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442947072:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203186.049530:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442947200:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203186.049556:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203186.049571:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203186/real 1788203186] req@ffff90234519c380 x1875064442947200/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203186.049585:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442947200/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203186.049594:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203186.049598:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203186.049602:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442947200:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203186.049609:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442947328:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203186.049630:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203186.049643:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203186/real 1788203186] req@ffff90234519f480 x1875064442947328/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203186.049655:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442947328/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203186.049663:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203186.049666:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203186.049670:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442947328:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203186.049676:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442947456:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203186.049696:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203186.049708:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203186/real 1788203186] req@ffff90234519dc00 x1875064442947456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203186.049720:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442947456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203186.049728:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203186.049732:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203186.049735:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442947456:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203186.881090:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203187.905086:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203188.929091:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203189.953119:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203190.977066:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203191.169029:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203191.169038:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203191.169042:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203191.169050:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2462 00000100:00080000:2.0:1788203191.169059:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203191.169065:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203191.169104:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442947584/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203191.169126:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203191.169129:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203191.169131:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203191.169135:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2462 00000100:00080000:2.0:1788203191.169138:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203191.169143:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203191.169153:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203191.169167:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442947712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203191.169175:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442947584:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203191.169180:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203191.169184:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203191.169186:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203191.169190:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2462 00000100:00080000:2.0:1788203191.169194:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203191.169198:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203191.169203:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203191.169215:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442947840/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203191.169222:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203191.169224:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203191.169225:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203191.169227:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2462 00000800:00000100:3.0:1788203191.169231:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788203191.169233:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203191.169238:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203191.169242:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203191.169250:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442947968/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203191.169273:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203191/real 1788203191] req@ffff90234598ce00 x1875064442947584/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203191.169287:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442947584/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203191.169294:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203191.169299:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203191.169302:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442947584:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203191.169321:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442947712:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203191.169335:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203191.169345:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203191/real 1788203191] req@ffff90234598f800 x1875064442947712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203191.169357:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442947712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203191.169364:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203191.169367:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203191.169371:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442947712:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203191.169376:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442947840:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203191.169395:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203191.169404:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203191/real 1788203191] req@ffff90234598dc00 x1875064442947840/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203191.169411:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442947840/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203191.169417:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203191.169420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203191.169424:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442947840:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203191.169429:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442947968:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203191.169447:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203191.169457:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203191/real 1788203191] req@ffff90234598f480 x1875064442947968/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203191.169468:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442947968/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203191.169473:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203191.169475:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203191.169477:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442947968:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203192.001100:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203193.025070:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203194.049042:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203195.073118:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203196.097080:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203196.289004:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203196.289016:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203196.289020:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203196.289031:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2467 00000100:00080000:2.0:1788203196.289041:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203196.289049:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203196.289084:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442948096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203196.289116:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203196.289120:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203196.289122:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203196.289126:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2467 00000100:00080000:2.0:1788203196.289130:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203196.289134:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203196.289143:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203196.289157:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442948224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203196.289168:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203196.289171:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203196.289173:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203196.289177:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2467 00000100:00080000:2.0:1788203196.289183:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203196.289186:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203196.289193:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203196.289201:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442948096:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203196.289207:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442948352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203196.289218:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203196.289222:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203196.289233:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203196.289237:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2467 00000100:00080000:2.0:1788203196.289243:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203196.289247:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203196.289252:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788203196.289267:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203196.289268:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442948480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203196.289308:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203196/real 1788203196] req@ffff90234598c380 x1875064442948096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203196.289341:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442948096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203196.289351:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203196.289356:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203196.289361:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442948096:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203196.289386:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442948224:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203196.289406:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203196.289417:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203196/real 1788203196] req@ffff90234598d180 x1875064442948224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203196.289429:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442948224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203196.289437:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203196.289441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203196.289444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442948224:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203196.289450:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442948352:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203196.289469:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203196.289492:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203196/real 1788203196] req@ffff90234598e300 x1875064442948352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203196.289506:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442948352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203196.289514:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203196.289518:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203196.289522:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442948352:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203196.289529:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442948480:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203196.289548:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203196.289560:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203196/real 1788203196] req@ffff90234598df80 x1875064442948480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203196.289575:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442948480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203196.289583:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203196.289587:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203196.289591:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442948480:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203197.121014:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203198.145037:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203199.169029:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203200.193036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203201.217017:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203201.408997:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203201.409016:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203201.409019:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203201.409028:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2472 00000100:00080000:0.0:1788203201.409036:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203201.409041:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203201.409071:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442948608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203201.409092:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203201.409095:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203201.409096:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203201.409098:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2472 00000100:00080000:0.0:1788203201.409100:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203201.409102:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203201.409108:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203201.409116:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442948736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203201.409122:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203201.409126:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203201.409127:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203201.409130:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2472 00000100:00080000:0.0:1788203201.409132:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788203201.409136:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442948608:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203201.409137:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203201.409143:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203201.409149:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442948864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203201.409156:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203201.409158:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203201.409159:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203201.409161:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2472 00000100:00080000:0.0:1788203201.409164:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203201.409166:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203201.409168:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203201.409175:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442948992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203201.409194:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203201.409233:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203201/real 1788203201] req@ffff9023450ba680 x1875064442948608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203201.409247:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442948608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203201.409255:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203201.409258:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203201.409260:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442948608:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203201.409265:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442948736:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203201.409278:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203201.409285:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203201/real 1788203201] req@ffff9023450b8e00 x1875064442948736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203201.409292:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442948736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203201.409297:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203201.409298:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203201.409300:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442948736:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203201.409316:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442948864:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203201.409326:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203201.409331:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203201/real 1788203201] req@ffff9023450bbb80 x1875064442948864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203201.409338:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442948864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203201.409342:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203201.409344:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203201.409345:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442948864:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203201.409348:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442948992:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203201.409357:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203201.409361:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203201/real 1788203201] req@ffff9023450bb480 x1875064442948992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203201.409372:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442948992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203201.409376:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203201.409378:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203201.409379:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442948992:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203202.241066:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203203.265027:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203204.289052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203205.313034:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203206.337026:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203206.528969:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203206.528977:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203206.528980:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203206.528988:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2478 00000100:00080000:0.0:1788203206.528996:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203206.529001:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203206.529033:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442949120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203206.529061:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203206.529063:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203206.529064:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203206.529067:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2478 00000100:00080000:0.0:1788203206.529069:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203206.529072:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203206.529079:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203206.529088:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442949248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203206.529096:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788203206.529098:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442949120:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203206.529099:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203206.529102:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203206.529109:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2478 00000100:00080000:0.0:1788203206.529111:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203206.529113:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203206.529116:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203206.529124:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442949376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203206.529130:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203206.529133:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203206.529134:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203206.529136:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2478 00000100:00080000:0.0:1788203206.529145:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203206.529147:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203206.529150:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203206.529156:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442949504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203206.529161:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203206.529196:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203206/real 1788203206] req@ffff9023450bb480 x1875064442949120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203206.529209:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442949120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203206.529216:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203206.529219:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203206.529221:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442949120:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203206.529237:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442949248:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203206.529250:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203206.529258:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203206/real 1788203206] req@ffff9023450bbb80 x1875064442949248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203206.529269:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442949248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203206.529274:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203206.529276:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203206.529277:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442949248:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203206.529281:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442949376:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203206.529290:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203206.529295:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203206/real 1788203206] req@ffff9023450b8e00 x1875064442949376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203206.529302:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442949376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203206.529306:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203206.529308:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203206.529309:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442949376:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203206.529313:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442949504:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203206.529321:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203206.529326:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203206/real 1788203206] req@ffff9023450ba680 x1875064442949504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203206.529332:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442949504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203206.529340:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203206.529342:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203206.529343:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442949504:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203207.361029:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203208.385095:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203209.409106:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203210.432980:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203211.457111:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203211.649045:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203211.649055:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203211.649060:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203211.649070:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2483 00000100:00080000:1.0:1788203211.649082:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203211.649089:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203211.649129:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442949632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203211.649156:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203211.649160:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203211.649162:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203211.649165:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2483 00000100:00080000:1.0:1788203211.649169:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203211.649173:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203211.649184:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203211.649198:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442949760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203211.649207:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203211.649211:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788203211.649212:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442949632:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203211.649213:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203211.649216:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2483 00000100:00080000:1.0:1788203211.649219:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203211.649222:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203211.649226:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203211.649234:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442949888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203211.649242:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203211.649245:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203211.649246:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203211.649250:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2483 00000100:00080000:1.0:1788203211.649252:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203211.649254:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203211.649258:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203211.649266:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442950016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203211.649288:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203211.649340:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203211/real 1788203211] req@ffff90234519f100 x1875064442949632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203211.649354:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442949632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203211.649361:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203211.649364:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203211.649366:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442949632:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203211.649390:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442949760:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203211.649407:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203211.649416:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203211/real 1788203211] req@ffff90234519dc00 x1875064442949760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203211.649424:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442949760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203211.649429:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203211.649431:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203211.649433:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442949760:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203211.649436:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442949888:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203211.649446:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203211.649451:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203211/real 1788203211] req@ffff90234519f480 x1875064442949888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203211.649458:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442949888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203211.649462:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203211.649464:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203211.649465:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442949888:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203211.649468:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442950016:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203211.649476:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203211.649481:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203211/real 1788203211] req@ffff90234519c380 x1875064442950016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203211.649488:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442950016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203211.649496:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203211.649498:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203211.649500:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442950016:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203212.481018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203213.505070:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203214.529047:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203215.553052:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203216.577035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203216.769049:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203216.769059:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203216.769064:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203216.769075:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2488 00000100:00080000:0.0:1788203216.769089:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203216.769097:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203216.769158:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442950144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203216.769191:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203216.769194:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203216.769195:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203216.769199:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2488 00000100:00080000:0.0:1788203216.769202:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203216.769205:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203216.769214:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203216.769225:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442950272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203216.769232:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203216.769236:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203216.769237:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203216.769239:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2488 00000100:00080000:0.0:1788203216.769241:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203216.769245:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203216.769249:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203216.769249:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442950144:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203216.769257:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442950400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203216.769263:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203216.769265:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203216.769265:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203216.769268:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2488 00000100:00080000:0.0:1788203216.769270:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203216.769273:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203216.769275:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203216.769282:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442950528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203216.769327:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203216.769376:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203216/real 1788203216] req@ffff9023450b8380 x1875064442950144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203216.769395:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442950144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203216.769406:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203216.769410:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203216.769414:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442950144:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203216.769421:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442950272:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203216.769440:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203216.769450:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203216/real 1788203216] req@ffff9023450b8a80 x1875064442950272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203216.769461:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442950272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203216.769467:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203216.769474:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203216.769476:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442950272:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203216.769498:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442950400:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203216.769512:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203216.769519:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203216/real 1788203216] req@ffff9023450bb800 x1875064442950400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203216.769526:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442950400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203216.769531:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203216.769556:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203216.769558:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442950400:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203216.769563:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442950528:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203216.769575:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203216.769582:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203216/real 1788203216] req@ffff9023450b9500 x1875064442950528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203216.769589:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442950528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203216.769595:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203216.769598:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203216.769601:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442950528:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203217.601050:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203218.625040:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203219.649039:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203220.673093:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203221.697027:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203221.888978:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203221.888987:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203221.888992:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203221.889002:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2493 00000100:00080000:1.0:1788203221.889013:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203221.889020:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203221.889068:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442950656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203221.889090:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203221.889093:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203221.889094:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203221.889097:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2493 00000100:00080000:1.0:1788203221.889099:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203221.889101:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203221.889110:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203221.889119:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442950784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203221.889130:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203221.889134:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203221.889136:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203221.889139:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2493 00000100:00080000:1.0:1788203221.889141:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203221.889145:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203221.889151:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203221.889154:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442950656:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203221.889162:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442950912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203221.889195:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203221.889198:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203221.889200:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203221.889206:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2493 00000100:00080000:1.0:1788203221.889210:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203221.889214:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203221.889219:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203221.889228:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442951040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203221.889246:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203221.889302:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203221/real 1788203221] req@ffff90234519ea00 x1875064442950656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203221.889323:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442950656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203221.889335:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203221.889339:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203221.889344:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442950656:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203221.889352:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442950784:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203221.889376:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203221.889388:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203221/real 1788203221] req@ffff90234519c700 x1875064442950784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203221.889400:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442950784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203221.889408:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203221.889411:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203221.889415:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442950784:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203221.889439:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442950912:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203221.889472:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203221.889484:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203221/real 1788203221] req@ffff90234519fb80 x1875064442950912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203221.889496:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442950912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203221.889509:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203221.889513:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203221.889516:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442950912:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203221.889523:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442951040:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203221.889542:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203221.889552:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203221/real 1788203221] req@ffff90234519df80 x1875064442951040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203221.889564:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442951040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203221.889571:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203221.889574:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203221.889577:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442951040:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203222.721032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203223.745046:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203224.769036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203225.793031:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203226.817034:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203227.009013:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203227.009023:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203227.009027:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203227.009037:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2498 00000100:00080000:2.0:1788203227.009046:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203227.009053:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203227.009083:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442951168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203227.009102:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203227.009105:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203227.009106:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203227.009109:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2498 00000100:00080000:2.0:1788203227.009111:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203227.009113:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203227.009121:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203227.009133:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442951296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203227.009144:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203227.009147:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788203227.009150:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442951168:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203227.009151:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203227.009158:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2498 00000100:00080000:2.0:1788203227.009162:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203227.009166:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203227.009171:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203227.009182:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442951424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203227.009189:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203227.009191:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203227.009192:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203227.009194:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2498 00000100:00080000:2.0:1788203227.009199:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203227.009201:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203227.009204:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203227.009208:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203227.009213:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442951552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203227.009243:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203227/real 1788203227] req@ffff90234598fb80 x1875064442951168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203227.009255:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442951168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203227.009262:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203227.009265:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203227.009267:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442951168:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203227.009282:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442951296:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203227.009295:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203227.009302:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203227/real 1788203227] req@ffff90234598ed80 x1875064442951296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203227.009309:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442951296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203227.009314:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203227.009316:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203227.009318:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442951296:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203227.009321:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442951424:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203227.009330:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203227.009335:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203227/real 1788203227] req@ffff90234598f100 x1875064442951424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203227.009342:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442951424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203227.009350:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203227.009353:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203227.009381:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442951424:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203227.009388:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442951552:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203227.009405:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203227.009415:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203227/real 1788203227] req@ffff90234598df80 x1875064442951552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203227.009425:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442951552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203227.009432:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203227.009434:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203227.009437:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442951552:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203227.841024:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203228.865068:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203229.889060:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203230.913053:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203231.937024:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203232.128939:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203232.128947:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203232.128950:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203232.128958:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2503 00000100:00080000:2.0:1788203232.128966:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203232.128971:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203232.129002:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442951680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203232.129027:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203232.129030:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203232.129032:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203232.129036:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2503 00000100:00080000:2.0:1788203232.129039:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203232.129041:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203232.129049:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203232.129059:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442951808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203232.129067:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203232.129069:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203232.129071:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788203232.129075:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442951680:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203232.129076:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2503 00000100:00080000:2.0:1788203232.129081:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203232.129084:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203232.129087:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203232.129097:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442951936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203232.129103:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203232.129106:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203232.129107:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203232.129110:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2503 00000100:00080000:2.0:1788203232.129112:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203232.129115:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203232.129118:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203232.129125:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442952064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203232.129138:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203232.129170:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203232/real 1788203232] req@ffff90234598e300 x1875064442951680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203232.129185:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442951680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203232.129194:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203232.129199:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203232.129205:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442951680:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203232.129228:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442951808:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203232.129263:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203232.129288:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203232/real 1788203232] req@ffff90234598d180 x1875064442951808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203232.129297:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442951808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203232.129303:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203232.129305:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203232.129308:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442951808:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203232.129313:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442951936:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203232.129331:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203232.129348:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203232/real 1788203232] req@ffff90234598c380 x1875064442951936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203232.129357:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442951936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203232.129365:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203232.129370:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203232.129373:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442951936:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203232.129382:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442952064:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203232.129403:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203232.129413:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203232/real 1788203232] req@ffff90234598f480 x1875064442952064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203232.129432:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442952064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203232.129439:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203232.129441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203232.129444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442952064:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203232.961024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203233.985055:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203235.009012:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203236.033044:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203237.057018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203237.248998:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203237.249006:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203237.249009:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203237.249018:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2508 00000100:00080000:1.0:1788203237.249029:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203237.249036:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203237.249080:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442952192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203237.249101:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203237.249104:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203237.249106:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203237.249110:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2508 00000100:00080000:1.0:1788203237.249114:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203237.249118:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203237.249127:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203237.249141:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442952320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203237.249147:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442952192:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203237.249151:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203237.249154:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203237.249155:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203237.249158:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2508 00000100:00080000:1.0:1788203237.249161:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203237.249164:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203237.249168:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203237.249175:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442952448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203237.249194:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203237.249197:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203237.249198:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203237.249202:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2508 00000100:00080000:1.0:1788203237.249204:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203237.249207:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203237.249211:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203237.249220:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442952576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203237.249259:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442952320:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203237.249285:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442952448:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203237.249297:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442952576:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203238.081040:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203239.105017:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203240.129163:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788203240.321155:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788203240.321197:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788203240.321211:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788203240.321251:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788203240.321259:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788203240.321267:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788203240.321356:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203237/real 1788203240] req@ffff90234519c000 x1875064442952320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203240.321383:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442952320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203240.321395:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203240.321401:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203240.321405:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442952320:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203240.321413:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203237/real 1788203240] req@ffff90234519e300 x1875064442952448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203240.321425:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442952448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203240.321434:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203240.321438:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203240.321441:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442952448:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203240.321447:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203237/real 1788203240] req@ffff90234519f800 x1875064442952576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203240.321457:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442952576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203240.321465:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203240.321468:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203240.321471:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442952576:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203240.321478:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203237/real 1788203240] req@ffff90234519ed80 x1875064442952192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203240.321491:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442952192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203240.321497:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203240.321501:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203240.321504:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442952192:192.168.201.130@tcp:250:kworker.0 00010000:00010000:2.0:1788203241.152985:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203242.176995:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203242.369000:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203242.369010:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203242.369013:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203242.369023:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2513 00000100:00080000:1.0:1788203242.369033:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203242.369039:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203242.369072:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442952704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203242.369113:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203242.369117:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203242.369119:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203242.369123:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2513 00000100:00080000:1.0:1788203242.369126:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203242.369129:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203242.369140:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203242.369152:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442952832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203242.369160:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203242.369163:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203242.369165:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203242.369166:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442952704:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203242.369169:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2513 00000100:00080000:1.0:1788203242.369174:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203242.369179:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203242.369183:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203242.369195:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442952960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203242.369203:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203242.369207:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203242.369209:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203242.369212:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2513 00000100:00080000:1.0:1788203242.369216:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203242.369219:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203242.369224:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203242.369233:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203242.369236:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442953088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203242.369295:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203242/real 1788203242] req@ffff90234519d880 x1875064442952704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203242.369331:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442952704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203242.369343:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203242.369346:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203242.369350:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442952704:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203242.369378:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442952832:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203242.369402:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203242.369415:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203242/real 1788203242] req@ffff90234519ed80 x1875064442952832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203242.369427:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442952832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203242.369435:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203242.369437:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203242.369441:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442952832:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203242.369446:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442952960:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203242.369465:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203242.369475:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203242/real 1788203242] req@ffff90234519f800 x1875064442952960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203242.369486:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442952960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203242.369500:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203242.369503:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203242.369506:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442952960:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203242.369512:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442953088:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203242.369531:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203242.369541:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203242/real 1788203242] req@ffff90234519e300 x1875064442953088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203242.369552:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442953088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203242.369560:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203242.369563:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203242.369566:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442953088:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203243.201049:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203244.225024:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203245.249101:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203246.273066:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203247.297059:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203247.489019:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203247.489029:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203247.489033:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203247.489044:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2519 00000100:00080000:0.0:1788203247.489053:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203247.489059:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203247.489093:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442953216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203247.489124:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203247.489127:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203247.489128:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203247.489132:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2519 00000100:00080000:0.0:1788203247.489135:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203247.489138:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203247.489147:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203247.489158:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442953344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203247.489167:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788203247.489170:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442953216:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203247.489171:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203247.489175:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203247.489181:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2519 00000100:00080000:0.0:1788203247.489184:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203247.489187:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203247.489190:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203247.489200:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442953472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203247.489208:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203247.489211:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203247.489212:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203247.489215:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2519 00000100:00080000:0.0:1788203247.489219:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203247.489222:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203247.489226:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203247.489229:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203247.489245:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442953600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203247.489278:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203247/real 1788203247] req@ffff9023450bad80 x1875064442953216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203247.489296:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442953216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203247.489306:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203247.489310:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203247.489314:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442953216:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203247.489336:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442953344:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203247.489355:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203247.489367:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203247/real 1788203247] req@ffff9023450b9c00 x1875064442953344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203247.489379:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442953344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203247.489388:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203247.489392:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203247.489395:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442953344:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203247.489400:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442953472:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203247.489416:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203247.489423:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203247/real 1788203247] req@ffff9023450b9f80 x1875064442953472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203247.489430:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442953472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203247.489435:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203247.489437:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203247.489440:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442953472:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203247.489444:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442953600:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203247.489460:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203247.489470:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203247/real 1788203247] req@ffff9023450b8700 x1875064442953600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203247.489481:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442953600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203247.489488:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203247.489491:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203247.489493:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442953600:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203248.321071:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203249.345043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203250.369116:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203251.393025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203252.417080:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203252.608979:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203252.608987:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203252.608990:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203252.608999:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2524 00000100:00080000:0.0:1788203252.609008:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203252.609013:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203252.609046:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442953728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203252.609068:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203252.609070:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203252.609071:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203252.609074:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2524 00000100:00080000:0.0:1788203252.609079:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203252.609081:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203252.609088:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203252.609097:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442953856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203252.609104:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203252.609106:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203252.609107:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203252.609109:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2524 00000100:00080000:0.0:1788203252.609111:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203252.609113:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203252.609116:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203252.609121:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442953728:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203252.609125:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442953984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203252.609135:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203252.609138:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203252.609139:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203252.609141:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2524 00000100:00080000:0.0:1788203252.609143:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203252.609145:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203252.609148:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203252.609154:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442954112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203252.609184:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203252.609220:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203252/real 1788203252] req@ffff9023450baa00 x1875064442953728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203252.609237:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442953728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203252.609247:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203252.609250:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203252.609253:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442953728:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203252.609261:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442953856:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203252.609280:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203252.609292:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203252/real 1788203252] req@ffff9023450b9180 x1875064442953856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203252.609302:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442953856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203252.609314:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203252.609317:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203252.609320:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442953856:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203252.609341:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442953984:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203252.609359:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203252.609368:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203252/real 1788203252] req@ffff9023450b8000 x1875064442953984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203252.609380:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442953984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203252.609387:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203252.609390:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203252.609393:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442953984:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203252.609397:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442954112:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203252.609413:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203252.609422:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203252/real 1788203252] req@ffff9023450ba300 x1875064442954112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203252.609432:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442954112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203252.609438:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203252.609441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203252.609444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442954112:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203253.441045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203254.465046:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203255.489051:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203256.513036:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203257.537016:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203257.728999:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203257.729008:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203257.729012:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203257.729023:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2529 00000100:00080000:2.0:1788203257.729033:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203257.729039:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203257.729076:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442954240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203257.729100:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203257.729104:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203257.729106:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203257.729110:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2529 00000100:00080000:2.0:1788203257.729114:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203257.729118:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203257.729127:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203257.729140:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442954240:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203257.729141:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442954368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203257.729155:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203257.729158:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203257.729160:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203257.729165:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2529 00000100:00080000:2.0:1788203257.729168:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203257.729172:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203257.729177:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203257.729188:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442954496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203257.729193:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788203257.729203:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203257.729207:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203257.729209:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203257.729213:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2529 00000100:00080000:2.0:1788203257.729217:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203257.729221:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203257.729225:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203257.729229:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203257/real 1788203257] req@ffff90234598dc00 x1875064442954240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203257.729236:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442954624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203257.729261:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442954240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203257.729273:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203257.729275:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203257.729278:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442954240:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203257.729295:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442954368:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203257.729312:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203257.729320:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203257/real 1788203257] req@ffff90234598f800 x1875064442954368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203257.729328:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442954368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203257.729333:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203257.729334:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203257.729337:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442954368:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203257.729340:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442954496:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203257.729356:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203257.729366:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203257/real 1788203257] req@ffff90234598ce00 x1875064442954496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203257.729377:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442954496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203257.729384:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203257.729387:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203257.729390:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442954496:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203257.729395:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442954624:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203257.729412:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203257.729420:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203257/real 1788203257] req@ffff90234598c700 x1875064442954624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203257.729426:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442954624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203257.729431:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203257.729433:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203257.729434:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442954624:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203258.561020:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203259.585068:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203260.609048:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203261.633070:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203262.657068:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203262.848948:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203262.848956:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203262.848959:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203262.848967:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2534 00000100:00080000:2.0:1788203262.848976:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203262.848987:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203262.849024:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442954752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203262.849058:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203262.849062:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203262.849063:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203262.849067:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2534 00000100:00080000:2.0:1788203262.849071:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203262.849075:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203262.849084:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203262.849098:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064442954880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203262.849106:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203262.849108:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203262.849110:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203262.849113:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2534 00000100:00080000:2.0:1788203262.849115:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203262.849117:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203262.849120:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203262.849129:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064442955008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203262.849139:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442954752:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203262.849161:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203262.849163:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203262.849164:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203262.849167:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2534 00000100:00080000:2.0:1788203262.849169:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203262.849172:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203262.849175:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203262.849184:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442955136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203262.849222:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203262.849273:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203262/real 1788203262] req@ffff90234598d880 x1875064442954752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203262.849293:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442954752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203262.849305:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203262.849310:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203262.849314:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442954752:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203262.849323:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442954880:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203262.849360:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203262.849383:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203262/real 1788203262] req@ffff90234598e680 x1875064442954880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203262.849397:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064442954880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203262.849405:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203262.849409:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203262.849411:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442954880:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203262.849437:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442955008:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203262.849453:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203262.849461:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203262/real 1788203262] req@ffff90234598ca80 x1875064442955008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203262.849469:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064442955008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203262.849474:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203262.849476:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203262.849478:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442955008:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203262.849483:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442955136:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203262.849492:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203262.849498:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203262/real 1788203262] req@ffff90234598c700 x1875064442955136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203262.849507:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442955136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203262.849524:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203262.849527:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203262.849530:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442955136:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203263.681060:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203264.705066:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203265.729050:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203266.753004:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203267.777044:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203267.968976:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203267.968985:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203267.968989:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203267.968997:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2539 00000100:00080000:2.0:1788203267.969006:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203267.969012:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203267.969049:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442955264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203267.969073:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203267.969076:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203267.969077:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203267.969080:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2539 00000100:00080000:2.0:1788203267.969082:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203267.969085:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203267.969093:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203267.969103:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442955392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203267.969111:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203267.969114:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203267.969116:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203267.969119:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2539 00000100:00080000:2.0:1788203267.969123:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788203267.969124:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442955264:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203267.969129:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203267.969135:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203267.969152:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442955520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203267.969161:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203267.969164:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203267.969166:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203267.969169:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2539 00000100:00080000:2.0:1788203267.969172:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203267.969174:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203267.969177:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203267.969185:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442955648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203267.969197:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203267.969246:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203267/real 1788203267] req@ffff90234598ce00 x1875064442955264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203267.969263:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442955264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203267.969272:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203267.969277:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203267.969281:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442955264:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203267.969289:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442955392:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203267.969310:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203267.969321:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203267/real 1788203267] req@ffff90234598f800 x1875064442955392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203267.969330:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442955392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203267.969336:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203267.969339:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203267.969342:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442955392:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203267.969361:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442955520:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203267.969373:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203267.969380:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203267/real 1788203267] req@ffff90234598dc00 x1875064442955520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203267.969388:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442955520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203267.969394:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203267.969396:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203267.969398:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442955520:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203267.969401:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442955648:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203267.969411:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203267.969417:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203267/real 1788203267] req@ffff90234598f480 x1875064442955648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203267.969426:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442955648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203267.969432:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203267.969434:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203267.969444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442955648:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203268.801038:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203269.825057:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203270.849028:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203271.872989:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203272.897051:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203273.088937:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203273.088945:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203273.088948:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203273.088957:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2544 00000100:00080000:2.0:1788203273.088966:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203273.088972:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203273.089005:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442955776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203273.089037:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203273.089040:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203273.089042:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203273.089045:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2544 00000100:00080000:2.0:1788203273.089047:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203273.089050:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203273.089057:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203273.089067:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442955904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203273.089079:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203273.089081:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203273.089083:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203273.089085:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2544 00000100:00080000:2.0:1788203273.089087:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203273.089092:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203273.089093:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442955776:192.168.201.130@tcp:250:kworker.0 00000008:00100000:2.0:1788203273.089096:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203273.089108:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442956032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203273.089119:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203273.089122:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203273.089123:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203273.089125:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2544 00000100:00080000:2.0:1788203273.089127:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203273.089130:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203273.089133:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203273.089140:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442956160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203273.089165:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203273.089214:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203273/real 1788203273] req@ffff90234598c380 x1875064442955776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203273.089231:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442955776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203273.089239:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203273.089242:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203273.089245:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442955776:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203273.089265:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442955904:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203273.089283:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203273.089292:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203273/real 1788203273] req@ffff90234598d180 x1875064442955904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203273.089299:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442955904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203273.089305:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203273.089307:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203273.089309:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442955904:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203273.089313:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442956032:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203273.089323:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203273.089329:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203273/real 1788203273] req@ffff90234598e300 x1875064442956032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203273.089336:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442956032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203273.089342:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203273.089344:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203273.089345:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442956032:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203273.089350:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442956160:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203273.089359:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203273.089365:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203273/real 1788203273] req@ffff90234598df80 x1875064442956160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203273.089372:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442956160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203273.089378:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203273.089379:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203273.089381:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442956160:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203273.921032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203274.945036:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203275.969017:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203276.993003:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203278.017039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203278.208987:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203278.208995:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203278.208999:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203278.209008:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2549 00000100:00080000:3.0:1788203278.209017:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203278.209022:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203278.209057:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442956288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203278.209079:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203278.209081:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203278.209083:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203278.209092:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2549 00000100:00080000:3.0:1788203278.209095:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203278.209099:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203278.209106:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203278.209116:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442956416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203278.209125:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788203278.209126:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442956288:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203278.209129:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203278.209132:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203278.209137:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2549 00000100:00080000:3.0:1788203278.209139:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203278.209142:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203278.209145:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203278.209154:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442956544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203278.209161:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203278.209164:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203278.209166:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203278.209168:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2549 00000100:00080000:3.0:1788203278.209171:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203278.209173:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203278.209176:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203278.209182:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442956672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203278.209190:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203278.209239:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203278/real 1788203278] req@ffff902352689500 x1875064442956288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203278.209256:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442956288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203278.209264:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203278.209267:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203278.209270:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442956288:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203278.209288:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442956416:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203278.209306:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203278.209315:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203278/real 1788203278] req@ffff902352688e00 x1875064442956416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203278.209323:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442956416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203278.209328:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203278.209330:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203278.209332:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442956416:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203278.209336:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442956544:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203278.209346:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203278.209355:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203278/real 1788203278] req@ffff90235268a680 x1875064442956544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203278.209368:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442956544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203278.209376:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203278.209380:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203278.209383:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442956544:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203278.209389:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442956672:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203278.209408:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203278.209417:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203278/real 1788203278] req@ffff902352689c00 x1875064442956672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203278.209430:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442956672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203278.209435:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203278.209437:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203278.209439:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442956672:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203279.041091:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203280.065076:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203281.089120:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203282.113066:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203283.137069:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203283.329063:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203283.329074:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203283.329078:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203283.329090:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2554 00000100:00080000:0.0:1788203283.329114:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203283.329123:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203283.329176:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442956800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203283.329212:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203283.329217:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203283.329219:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203283.329224:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2554 00000100:00080000:0.0:1788203283.329229:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203283.329234:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203283.329248:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203283.329264:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442956800:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203283.329267:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442956928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203283.329280:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203283.329283:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203283.329285:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203283.329289:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2554 00000100:00080000:0.0:1788203283.329292:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203283.329295:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203283.329301:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203283.329311:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442957056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203283.329319:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203283.329324:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203283.329326:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203283.329330:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2554 00000100:00080000:0.0:1788203283.329334:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203283.329343:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:1.0:1788203283.329343:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:0.0:1788203283.329352:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203283.329366:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442957184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203283.329428:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203283/real 1788203283] req@ffff9023450ba300 x1875064442956800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203283.329447:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442956800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203283.329467:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203283.329470:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203283.329474:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442956800:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203283.329497:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442956928:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203283.329515:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203283.329524:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203283/real 1788203283] req@ffff9023450b8000 x1875064442956928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203283.329537:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442956928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203283.329543:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203283.329545:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203283.329547:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442956928:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203283.329551:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442957056:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203283.329563:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203283.329569:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203283/real 1788203283] req@ffff9023450b9180 x1875064442957056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203283.329577:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442957056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203283.329582:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203283.329584:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203283.329586:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442957056:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203283.329589:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442957184:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203283.329600:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203283.329610:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203283/real 1788203283] req@ffff9023450baa00 x1875064442957184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203283.329622:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442957184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203283.329637:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203283.329641:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203283.329644:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442957184:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203284.161050:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203285.185046:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203286.209013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203287.233093:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203288.257039:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203288.449006:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203288.449017:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203288.449022:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203288.449033:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2559 00000100:00080000:0.0:1788203288.449053:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203288.449061:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203288.449113:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442957312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203288.449151:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203288.449155:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203288.449157:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203288.449163:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2559 00000100:00080000:0.0:1788203288.449167:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203288.449172:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203288.449183:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203288.449204:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442957440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203288.449218:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203288.449222:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203288.449224:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203288.449233:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2559 00000100:00100000:1.0:1788203288.449234:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442957312:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203288.449240:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203288.449251:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203288.449257:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203288.449270:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442957568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203288.449279:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203288.449283:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203288.449285:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203288.449304:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2559 00000100:00080000:0.0:1788203288.449310:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203288.449314:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203288.449320:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788203288.449327:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203288.449332:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442957696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203288.449375:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203288/real 1788203288] req@ffff9023450b8700 x1875064442957312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203288.449391:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442957312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203288.449400:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203288.449403:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203288.449407:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442957312:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203288.449427:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442957440:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203288.449445:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203288.449454:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203288/real 1788203288] req@ffff9023450b9f80 x1875064442957440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203288.449462:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442957440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203288.449471:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203288.449474:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203288.449476:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442957440:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203288.449480:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442957568:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203288.449491:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203288.449497:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203288/real 1788203288] req@ffff9023450b9c00 x1875064442957568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203288.449504:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442957568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203288.449510:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203288.449512:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203288.449513:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442957568:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203288.449517:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442957696:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203288.449527:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203288.449546:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203288/real 1788203288] req@ffff9023450bad80 x1875064442957696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203288.449554:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442957696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203288.449559:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203288.449561:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203288.449563:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442957696:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203289.281080:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203290.305020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203291.329043:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203292.353026:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203293.377010:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203293.568989:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203293.569003:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203293.569007:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203293.569016:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2565 00000100:00080000:3.0:1788203293.569025:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203293.569031:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203293.569063:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442957824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203293.569084:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203293.569087:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203293.569088:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203293.569093:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2565 00000100:00080000:3.0:1788203293.569096:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203293.569100:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203293.569111:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203293.569125:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442957824:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203293.569136:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442957952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203293.569151:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203293.569153:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203293.569155:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203293.569158:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2565 00000100:00080000:3.0:1788203293.569162:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203293.569165:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203293.569169:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203293.569184:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442958080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203293.569193:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788203293.569196:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203293.569218:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203293.569220:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203293.569224:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2565 00000100:00080000:3.0:1788203293.569228:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203293.569232:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203293.569237:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203293.569257:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442958208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203293.569278:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203293/real 1788203293] req@ffff902352689180 x1875064442957824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203293.569296:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442957824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203293.569305:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203293.569310:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203293.569315:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442957824:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203293.569339:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442957952:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203293.569364:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203293.569375:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203293/real 1788203293] req@ffff902352688700 x1875064442957952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203293.569384:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442957952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203293.569390:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203293.569393:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203293.569396:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442957952:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203293.569403:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442958080:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203293.569425:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203293.569438:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203293/real 1788203293] req@ffff90235268bb80 x1875064442958080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203293.569451:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442958080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203293.569461:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203293.569464:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203293.569468:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442958080:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203293.569475:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442958208:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203293.569492:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203293.569500:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203293/real 1788203293] req@ffff902352689880 x1875064442958208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203293.569508:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442958208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203293.569514:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203293.569516:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203293.569518:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442958208:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203294.401028:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203295.425014:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203296.449031:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203297.473023:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203298.497064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203298.689044:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203298.689053:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203298.689056:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203298.689065:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2570 00000100:00080000:2.0:1788203298.689075:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203298.689081:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203298.689138:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442958336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203298.689166:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203298.689169:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203298.689170:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203298.689173:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2570 00000100:00080000:2.0:1788203298.689176:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203298.689179:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203298.689188:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203298.689198:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442958464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203298.689207:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203298.689209:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203298.689211:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788203298.689215:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442958336:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203298.689217:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2570 00000100:00080000:2.0:1788203298.689222:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203298.689226:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203298.689229:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203298.689238:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442958592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203298.689244:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203298.689247:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203298.689249:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203298.689251:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2570 00000100:00080000:2.0:1788203298.689255:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203298.689257:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203298.689260:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203298.689268:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442958720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203298.689277:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203298.689312:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203298/real 1788203298] req@ffff90234598f100 x1875064442958336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203298.689324:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442958336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203298.689332:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203298.689335:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203298.689338:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442958336:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203298.689356:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442958464:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203298.689378:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203298.689391:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203298/real 1788203298] req@ffff90234598ed80 x1875064442958464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203298.689405:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442958464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203298.689414:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203298.689416:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203298.689419:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442958464:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203298.689423:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442958592:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203298.689438:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203298.689446:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203298/real 1788203298] req@ffff90234598fb80 x1875064442958592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203298.689454:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442958592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203298.689463:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203298.689465:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203298.689467:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442958592:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203298.689470:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442958720:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203298.689479:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203298.689485:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203298/real 1788203298] req@ffff90234598df80 x1875064442958720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203298.689492:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442958720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203298.689497:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203298.689499:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203298.689501:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442958720:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203299.521062:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203300.545096:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203301.569085:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203302.593106:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203303.617091:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203303.809012:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203303.809041:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203303.809046:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203303.809058:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2575 00000100:00080000:0.0:1788203303.809071:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203303.809082:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203303.809134:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442958848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203303.809170:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203303.809175:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203303.809177:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203303.809186:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2575 00000100:00080000:0.0:1788203303.809193:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203303.809200:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203303.809212:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442958848:192.168.201.130@tcp:250:kworker.0 00000008:00100000:0.0:1788203303.809215:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203303.809232:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442958976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203303.809244:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203303.809249:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203303.809251:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203303.809256:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2575 00000100:00080000:0.0:1788203303.809261:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203303.809266:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203303.809273:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203303.809289:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442959104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203303.809297:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203303.809300:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203303.809303:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203303.809306:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2575 00000100:00080000:0.0:1788203303.809309:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203303.809313:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203303.809318:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203303.809327:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442959232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203303.809416:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442958976:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203303.809451:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442959104:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203303.809466:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442959232:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203304.641030:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203305.665002:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203306.689014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788203306.881097:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:0.0:1788203306.881139:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788203306.881157:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788203306.881190:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788203306.881199:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788203306.881208:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788203306.881243:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203303/real 1788203306] req@ffff9023450bb800 x1875064442958976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203306.881270:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442958976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203306.881283:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203306.881288:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203306.881293:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442958976:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203306.881310:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203303/real 1788203306] req@ffff9023450b8a80 x1875064442959104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203306.881325:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442959104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203306.881335:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203306.881340:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203306.881343:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442959104:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203306.881351:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203303/real 1788203306] req@ffff9023450b8380 x1875064442959232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203306.881365:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442959232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203306.881372:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203306.881375:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203306.881378:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442959232:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203306.881385:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203303/real 1788203306] req@ffff9023450b9500 x1875064442958848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203306.881397:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442958848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203306.881404:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203306.881408:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203306.881410:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442958848:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788203307.713023:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203308.737042:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203308.929006:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203308.929017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203308.929021:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203308.929032:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2580 00000100:00080000:3.0:1788203308.929043:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203308.929051:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203308.929083:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442959360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203308.929127:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203308.929131:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203308.929133:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203308.929138:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2580 00000100:00080000:3.0:1788203308.929142:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203308.929147:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203308.929156:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203308.929171:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442959488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203308.929181:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203308.929185:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203308.929187:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203308.929191:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2580 00000100:00080000:3.0:1788203308.929196:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203308.929201:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203308.929207:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203308.929220:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442959360:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203308.929223:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442959616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203308.929239:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203308.929242:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203308.929244:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203308.929246:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2580 00000100:00080000:3.0:1788203308.929266:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203308.929271:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203308.929277:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203308.929291:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442959744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203308.929318:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203308.929392:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203308/real 1788203308] req@ffff902352688380 x1875064442959360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203308.929414:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442959360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203308.929425:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203308.929430:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203308.929435:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442959360:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203308.929444:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442959488:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203308.929472:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203308.929485:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203308/real 1788203308] req@ffff90235268ad80 x1875064442959488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203308.929498:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442959488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203308.929506:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203308.929509:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203308.929512:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442959488:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203308.929539:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442959616:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203308.929563:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203308.929575:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203308/real 1788203308] req@ffff90235268b480 x1875064442959616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203308.929588:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442959616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203308.929603:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203308.929607:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203308.929610:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442959616:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203308.929617:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442959744:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203308.929639:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203308.929652:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203308/real 1788203308] req@ffff902352688a80 x1875064442959744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203308.929663:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442959744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203308.929687:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203308.929692:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203308.929695:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442959744:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203309.761008:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203310.785109:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203311.809155:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203312.833057:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203313.857016:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203314.048955:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203314.048967:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203314.048972:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203314.048982:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2585 00000100:00080000:3.0:1788203314.048995:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203314.049003:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203314.049049:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064442959872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203314.049076:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203314.049084:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203314.049085:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203314.049089:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2585 00000100:00080000:3.0:1788203314.049092:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203314.049095:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203314.049105:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203314.049118:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064442960000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203314.049126:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203314.049129:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203314.049130:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203314.049133:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2585 00000100:00080000:3.0:1788203314.049136:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788203314.049139:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442959872:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203314.049142:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203314.049150:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203314.049158:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064442960128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203314.049164:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203314.049167:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203314.049168:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203314.049171:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2585 00000100:00080000:3.0:1788203314.049173:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203314.049176:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203314.049179:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203314.049187:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064442960256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203314.049227:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203314.049280:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203314/real 1788203314] req@ffff90235268aa00 x1875064442959872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203314.049299:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064442959872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203314.049313:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203314.049316:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203314.049320:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442959872:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203314.049342:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442960000:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203314.049360:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203314.049376:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203314/real 1788203314] req@ffff902352688a80 x1875064442960000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203314.049385:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064442960000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203314.049391:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203314.049393:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203314.049395:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442960000:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203314.049399:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442960128:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203314.049412:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203314.049419:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203314/real 1788203314] req@ffff90235268b480 x1875064442960128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203314.049426:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064442960128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203314.049432:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203314.049434:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203314.049435:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442960128:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203314.049440:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442960256:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203314.049454:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203314.049465:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203314/real 1788203314] req@ffff90235268ad80 x1875064442960256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203314.049478:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064442960256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203314.049486:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203314.049490:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203314.049494:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442960256:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203314.881124:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203315.905080:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203316.929042:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203317.953047:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203318.977050:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203319.168979:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203319.168988:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203319.168992:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203319.169001:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2590 00000100:00080000:1.0:1788203319.169010:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203319.169016:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203319.169050:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442960384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203319.169073:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203319.169076:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203319.169078:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203319.169081:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2590 00000100:00080000:1.0:1788203319.169084:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203319.169086:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203319.169095:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203319.169104:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442960512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203319.169112:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203319.169116:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203319.169118:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203319.169118:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442960384:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203319.169124:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2590 00000100:00080000:1.0:1788203319.169130:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203319.169133:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203319.169135:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203319.169145:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442960640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203319.169152:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203319.169155:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203319.169156:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203319.169158:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2590 00000100:00080000:1.0:1788203319.169160:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203319.169163:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203319.169166:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203319.169175:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442960768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203319.169184:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203319.169225:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203319/real 1788203319] req@ffff90234519c000 x1875064442960384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203319.169240:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442960384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203319.169249:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203319.169252:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203319.169255:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442960384:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203319.169271:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442960512:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203319.169286:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203319.169294:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203319/real 1788203319] req@ffff90234519df80 x1875064442960512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203319.169302:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442960512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203319.169308:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203319.169310:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203319.169312:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442960512:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203319.169316:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442960640:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203319.169326:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203319.169332:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203319/real 1788203319] req@ffff90234519fb80 x1875064442960640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203319.169340:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442960640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203319.169345:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203319.169346:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203319.169348:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442960640:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203319.169351:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442960768:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203319.169360:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203319.169366:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203319/real 1788203319] req@ffff90234519c700 x1875064442960768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203319.169374:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442960768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203319.169378:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203319.169380:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203319.169382:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442960768:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203320.001109:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203321.025088:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203322.049091:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203323.073077:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203324.097097:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203324.289048:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203324.289060:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203324.289064:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203324.289075:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2595 00000100:00080000:1.0:1788203324.289095:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203324.289102:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203324.289149:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442960896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203324.289180:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203324.289183:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203324.289185:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203324.289188:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2595 00000100:00080000:1.0:1788203324.289190:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203324.289193:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203324.289203:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203324.289214:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442961024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203324.289222:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203324.289225:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203324.289228:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203324.289231:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2595 00000100:00080000:1.0:1788203324.289234:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788203324.289238:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442960896:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203324.289240:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203324.289247:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203324.289256:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442961152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203324.289264:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203324.289267:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203324.289274:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203324.289277:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2595 00000100:00080000:1.0:1788203324.289286:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203324.289290:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203324.289296:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203324.289304:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442961280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203324.289327:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203324.289393:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203324/real 1788203324] req@ffff90234519ea00 x1875064442960896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203324.289411:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442960896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203324.289428:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203324.289432:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203324.289435:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442960896:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203324.289442:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442961024:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203324.289462:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203324.289472:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203324/real 1788203324] req@ffff90234519c380 x1875064442961024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203324.289480:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442961024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203324.289486:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203324.289488:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203324.289491:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442961024:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203324.289514:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442961152:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203324.289536:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203324.289548:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203324/real 1788203324] req@ffff90234519f480 x1875064442961152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203324.289560:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442961152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203324.289570:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203324.289573:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203324.289577:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442961152:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203324.289584:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442961280:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203324.289605:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203324.289616:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203324/real 1788203324] req@ffff90234519dc00 x1875064442961280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203324.289629:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442961280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203324.289635:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203324.289638:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203324.289640:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442961280:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203325.121006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203326.145092:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203327.169133:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203328.193059:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203329.217062:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203329.409020:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203329.409029:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203329.409032:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203329.409040:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2600 00000100:00080000:3.0:1788203329.409050:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203329.409056:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203329.409092:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064442961408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203329.409120:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203329.409123:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203329.409124:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203329.409127:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2600 00000100:00080000:3.0:1788203329.409130:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203329.409133:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203329.409143:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203329.409157:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064442961536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203329.409164:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442961408:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203329.409175:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203329.409180:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203329.409183:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203329.409189:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2600 00000100:00080000:3.0:1788203329.409193:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203329.409197:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203329.409201:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203329.409211:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064442961664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203329.409218:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203329.409221:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203329.409222:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788203329.409227:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788203329.409227:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2600 00000100:00080000:3.0:1788203329.409234:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203329.409236:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203329.409240:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203329.409248:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064442961792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203329.409276:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203329/real 1788203329] req@ffff902352688380 x1875064442961408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203329.409297:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064442961408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203329.409306:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203329.409308:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203329.409311:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442961408:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203329.409329:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442961536:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203329.409344:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203329.409354:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203329/real 1788203329] req@ffff902352689880 x1875064442961536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203329.409362:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064442961536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203329.409368:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203329.409370:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203329.409372:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442961536:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203329.409376:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442961664:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203329.409387:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203329.409393:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203329/real 1788203329] req@ffff90235268bb80 x1875064442961664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203329.409400:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064442961664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203329.409406:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203329.409407:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203329.409409:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442961664:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203329.409413:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442961792:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203329.409423:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203329.409428:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203329/real 1788203329] req@ffff902352688700 x1875064442961792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203329.409435:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064442961792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203329.409441:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203329.409442:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203329.409444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442961792:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203330.241073:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203331.265076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203332.289078:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203333.313033:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203334.337093:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203334.529021:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203334.529033:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203334.529038:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203334.529049:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2606 00000100:00080000:3.0:1788203334.529061:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203334.529069:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203334.529109:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064442961920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203334.529138:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203334.529143:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203334.529146:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203334.529151:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2606 00000100:00080000:3.0:1788203334.529156:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203334.529162:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203334.529167:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442961920:192.168.201.130@tcp:250:kworker.0 00000008:00100000:3.0:1788203334.529174:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203334.529194:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442962048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203334.529207:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203334.529211:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203334.529214:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203334.529217:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2606 00000100:00080000:3.0:1788203334.529222:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203334.529226:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:1.0:1788203334.529232:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:3.0:1788203334.529232:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203334.529259:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442962176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203334.529272:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203334.529276:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203334.529278:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788203334.529284:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203334/real 1788203334] req@ffff902352689180 x1875064442961920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203334.529287:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2606 00000100:00080000:3.0:1788203334.529295:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203334.529305:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203334.529312:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203334.529317:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064442961920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203334.529329:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442962304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203334.529333:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203334.529345:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203334.529353:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442961920:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203334.529380:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442962048:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203334.529410:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203334.529425:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203334/real 1788203334] req@ffff902352689c00 x1875064442962048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203334.529438:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442962048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203334.529446:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203334.529450:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203334.529453:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442962048:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203334.529459:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442962176:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203334.529477:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203334.529487:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203334/real 1788203334] req@ffff90235268a680 x1875064442962176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203334.529514:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442962176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203334.529522:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203334.529526:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203334.529529:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442962176:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203334.529535:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442962304:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203334.529551:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203334.529561:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203334/real 1788203334] req@ffff902352688e00 x1875064442962304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203334.529578:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442962304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203334.529585:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203334.529589:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203334.529592:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442962304:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203335.361027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203336.385041:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203337.409009:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203338.433066:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203339.457070:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203339.649061:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203339.649070:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203339.649075:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203339.649086:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2611 00000100:00080000:3.0:1788203339.649098:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203339.649106:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203339.649145:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442962432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203339.649183:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203339.649192:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203339.649195:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203339.649201:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2611 00000100:00080000:3.0:1788203339.649206:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203339.649211:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203339.649222:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203339.649238:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442962560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203339.649246:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442962432:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203339.649258:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203339.649263:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203339.649265:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203339.649271:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2611 00000100:00080000:3.0:1788203339.649275:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203339.649280:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203339.649285:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203339.649299:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442962688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203339.649308:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203339.649342:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203339/real 1788203339] req@ffff902352689500 x1875064442962432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203339.649357:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442962432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203339.649365:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203339.649369:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203339.649376:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442962432:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203339.649378:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203339.649384:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203339.649387:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203339.649393:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2611 00000100:00080000:3.0:1788203339.649398:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203339.649423:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203339.649429:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203339.649443:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442962816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203339.649446:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442962560:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203339.649478:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203339.649491:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203339/real 1788203339] req@ffff90235268b800 x1875064442962560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203339.649500:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442962560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203339.649507:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203339.649510:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203339.649514:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442962560:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203339.649519:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442962688:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203339.649533:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203339.649546:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203339/real 1788203339] req@ffff90235268a300 x1875064442962688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203339.649558:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442962688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203339.649568:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203339.649572:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203339.649575:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442962688:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203339.649588:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442962816:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203339.649604:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203339.649614:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203339/real 1788203339] req@ffff90235268b100 x1875064442962816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203339.649626:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442962816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203339.649644:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203339.649648:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203339.649653:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442962816:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203340.481049:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203341.505099:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203342.529028:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203343.553068:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203344.577062:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203344.768993:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203344.769003:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203344.769009:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203344.769023:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2616 00000100:00080000:3.0:1788203344.769036:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203344.769045:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203344.769104:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064442962944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203344.769142:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203344.769145:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203344.769147:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203344.769152:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2616 00000100:00080000:3.0:1788203344.769157:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203344.769164:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203344.769180:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203344.769201:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442962944:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203344.769205:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064442963072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203344.769229:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203344.769233:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203344.769236:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203344.769242:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2616 00000100:00080000:3.0:1788203344.769248:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203344.769255:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203344.769263:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203344.769276:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064442963200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203344.769285:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788203344.769303:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203344.769307:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203344.769310:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203344.769329:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2616 00000100:00080000:3.0:1788203344.769333:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203344.769337:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203344.769342:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203344.769354:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064442963328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203344.769361:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203344/real 1788203344] req@ffff902352688000 x1875064442962944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203344.769392:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064442962944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203344.769405:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203344.769411:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203344.769415:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442962944:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203344.769444:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442963072:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203344.769473:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203344.769488:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203344/real 1788203344] req@ffff90235268b100 x1875064442963072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203344.769502:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064442963072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203344.769511:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203344.769515:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203344.769518:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442963072:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203344.769526:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442963200:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203344.769548:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203344.769561:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203344/real 1788203344] req@ffff90235268a300 x1875064442963200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203344.769573:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064442963200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203344.769582:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203344.769586:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203344.769589:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442963200:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203344.769596:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442963328:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203344.769616:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203344.769628:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203344/real 1788203344] req@ffff90235268b800 x1875064442963328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203344.769641:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064442963328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203344.769649:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203344.769653:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203344.769657:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442963328:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203345.601103:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203346.625113:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203347.649121:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203348.673118:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203349.697098:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203349.889055:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203349.889064:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203349.889069:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203349.889079:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2621 00000100:00080000:0.0:1788203349.889090:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203349.889098:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203349.889139:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442963456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203349.889166:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203349.889169:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203349.889171:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203349.889174:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2621 00000100:00080000:0.0:1788203349.889177:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203349.889180:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203349.889189:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203349.889199:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442963584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203349.889208:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788203349.889211:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442963456:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203349.889212:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203349.889215:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203349.889239:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2621 00000100:00080000:0.0:1788203349.889242:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203349.889246:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203349.889250:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203349.889261:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442963712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203349.889269:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203349.889271:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203349.889272:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203349.889276:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2621 00000100:00080000:0.0:1788203349.889279:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203349.889286:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203349.889290:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788203349.889299:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203349.889303:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442963840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203349.889347:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203349/real 1788203349] req@ffff9023450ba680 x1875064442963456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203349.889369:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442963456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203349.889381:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203349.889384:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203349.889387:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442963456:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203349.889406:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442963584:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203349.889426:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203349.889435:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203349/real 1788203349] req@ffff9023450b8e00 x1875064442963584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203349.889444:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442963584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203349.889454:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203349.889456:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203349.889459:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442963584:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203349.889462:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442963712:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203349.889473:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203349.889479:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203349/real 1788203349] req@ffff9023450bbb80 x1875064442963712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203349.889487:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442963712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203349.889492:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203349.889494:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203349.889496:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442963712:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203349.889500:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442963840:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203349.889510:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203349.889516:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203349/real 1788203349] req@ffff9023450bb480 x1875064442963840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203349.889523:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442963840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203349.889538:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203349.889540:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203349.889546:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442963840:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203350.721041:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203351.745069:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203352.769043:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203353.793055:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203354.817016:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203355.009014:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203355.009022:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203355.009026:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203355.009034:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2626 00000100:00080000:2.0:1788203355.009044:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203355.009050:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203355.009083:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442963968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203355.009117:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203355.009120:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203355.009122:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203355.009126:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2626 00000100:00080000:2.0:1788203355.009130:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203355.009149:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203355.009160:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203355.009174:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442964096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203355.009184:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203355.009187:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203355.009189:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203355.009192:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2626 00000100:00100000:1.0:1788203355.009195:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442963968:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203355.009196:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203355.009200:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203355.009204:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203355.009212:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442964224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203355.009221:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203355.009224:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203355.009226:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203355.009229:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2626 00000100:00080000:2.0:1788203355.009232:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203355.009235:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203355.009239:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203355.009246:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442964352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203355.009256:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203355.009309:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203355/real 1788203355] req@ffff90234598e300 x1875064442963968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203355.009328:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442963968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203355.009349:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203355.009352:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203355.009358:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442963968:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203355.009364:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442964096:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203355.009383:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203355.009392:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203355/real 1788203355] req@ffff90234598d180 x1875064442964096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203355.009400:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442964096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203355.009406:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203355.009408:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203355.009410:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442964096:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203355.009428:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442964224:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203355.009439:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203355.009445:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203355/real 1788203355] req@ffff90234598c380 x1875064442964224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203355.009452:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442964224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203355.009458:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203355.009460:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203355.009462:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442964224:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203355.009466:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442964352:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203355.009475:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203355.009481:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203355/real 1788203355] req@ffff90234598f480 x1875064442964352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203355.009489:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442964352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203355.009494:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203355.009496:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203355.009498:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442964352:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203355.841057:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203356.865074:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203357.889047:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203358.913055:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203359.937077:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203360.129033:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203360.129056:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203360.129060:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203360.129071:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2631 00000100:00080000:1.0:1788203360.129084:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203360.129091:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203360.129137:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442964480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203360.129170:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203360.129174:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203360.129177:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203360.129181:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2631 00000100:00080000:1.0:1788203360.129185:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203360.129190:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203360.129200:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203360.129219:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442964608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203360.129224:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442964480:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203360.129239:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203360.129244:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203360.129246:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203360.129259:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2631 00000100:00080000:1.0:1788203360.129264:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203360.129269:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203360.129275:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203360.129288:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442964736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203360.129299:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788203360.129306:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788203360.129307:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203360.129311:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203360.129317:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2631 00000100:00080000:1.0:1788203360.129321:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203360.129326:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203360.129332:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203360.129344:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442964864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203360.129360:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203360/real 1788203360] req@ffff90234519f100 x1875064442964480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203360.129383:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442964480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203360.129396:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203360.129401:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203360.129405:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442964480:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203360.129432:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442964608:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203360.129454:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203360.129467:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203360/real 1788203360] req@ffff90234519dc00 x1875064442964608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203360.129480:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442964608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203360.129487:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203360.129489:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203360.129492:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442964608:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203360.129496:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442964736:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203360.129511:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203360.129519:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203360/real 1788203360] req@ffff90234519f480 x1875064442964736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203360.129526:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442964736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203360.129532:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203360.129533:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203360.129535:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442964736:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203360.129539:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442964864:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203360.129549:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203360.129554:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203360/real 1788203360] req@ffff90234519c380 x1875064442964864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203360.129562:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442964864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203360.129567:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203360.129569:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203360.129571:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442964864:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203360.961053:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203361.985076:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203363.009062:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203364.033089:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203365.057046:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203365.249034:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203365.249046:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203365.249051:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203365.249062:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2636 00000100:00080000:3.0:1788203365.249073:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203365.249079:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203365.249118:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064442964992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203365.249148:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203365.249152:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203365.249154:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203365.249159:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2636 00000100:00080000:3.0:1788203365.249163:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203365.249168:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203365.249179:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203365.249184:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442964992:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203365.249196:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064442965120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203365.249210:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203365.249214:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203365.249216:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203365.249220:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2636 00000100:00080000:3.0:1788203365.249240:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788203365.249249:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788203365.249249:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203365.249258:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203365.249274:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064442965248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203365.249285:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788203365.249291:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203365/real 1788203365] req@ffff902352689500 x1875064442964992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203365.249291:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203365.249295:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203365.249306:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2636 00000100:00080000:3.0:1788203365.249312:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203365.249318:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064442964992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203365.249319:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203365.249329:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1788203365.249332:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203365.249344:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203365.249347:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442964992:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203365.249358:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064442965376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203365.249375:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442965120:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203365.249394:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203365.249407:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203365/real 1788203365] req@ffff902352688e00 x1875064442965120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203365.249420:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064442965120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203365.249436:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203365.249439:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203365.249443:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442965120:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203365.249450:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442965248:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203365.249471:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203365.249482:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203365/real 1788203365] req@ffff90235268a680 x1875064442965248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203365.249491:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064442965248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203365.249497:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203365.249499:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203365.249502:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442965248:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203365.249505:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442965376:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203365.249519:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203365.249525:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203365/real 1788203365] req@ffff902352689c00 x1875064442965376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203365.249533:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064442965376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203365.249537:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203365.249539:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203365.249542:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442965376:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203366.081133:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203367.105086:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203368.129084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203369.153051:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203370.177075:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203370.369068:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203370.369080:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203370.369084:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203370.369096:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2641 00000100:00080000:0.0:1788203370.369115:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203370.369122:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203370.369174:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442965504/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203370.369207:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203370.369212:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203370.369214:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203370.369221:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2641 00000100:00080000:0.0:1788203370.369226:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203370.369231:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203370.369243:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203370.369245:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442965504:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203370.369259:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442965632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203370.369270:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203370.369272:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203370.369274:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203370.369278:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2641 00000100:00080000:0.0:1788203370.369280:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203370.369284:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203370.369287:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203370.369296:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442965760/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203370.369303:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203370.369306:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203370.369308:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203370.369311:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2641 00000100:00080000:0.0:1788203370.369313:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203370.369315:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203370.369318:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203370.369325:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442965888/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203370.369367:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442965632:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203370.369401:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442965760:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203370.369448:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442965888:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203371.201034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203372.225012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203373.249199:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788203373.441136:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:1.0:1788203373.441185:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788203373.441201:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788203373.441243:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788203373.441254:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788203373.441266:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:2.0:1788203373.441290:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203370/real 1788203373] req@ffff9023450bbb80 x1875064442965632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203373.441324:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442965632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203373.441337:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203373.441342:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203373.441347:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442965632:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203373.441355:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203370/real 1788203373] req@ffff9023450b8e00 x1875064442965760/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203373.441367:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442965760/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203373.441376:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203373.441380:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203373.441383:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442965760:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203373.441390:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203370/real 1788203373] req@ffff9023450ba680 x1875064442965888/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203373.441402:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442965888/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203373.441409:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203373.441413:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203373.441416:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442965888:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203373.441422:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203370/real 1788203373] req@ffff9023450bb480 x1875064442965504/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203373.441436:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442965504/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203373.441444:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203373.441447:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203373.441451:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442965504:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788203374.273016:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203375.297016:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203375.489026:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203375.489035:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203375.489039:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203375.489048:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2647 00000100:00080000:0.0:1788203375.489059:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203375.489066:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203375.489108:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442966016/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203375.489159:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203375.489163:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203375.489164:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203375.489169:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2647 00000100:00080000:0.0:1788203375.489175:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203375.489180:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203375.489192:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203375.489213:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442966016:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203375.489216:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442966144/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203375.489245:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203375.489249:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203375.489252:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203375.489258:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2647 00000100:00080000:0.0:1788203375.489263:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203375.489268:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203375.489275:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203375.489293:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442966272/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203375.489305:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203375.489309:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203375.489311:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203375.489315:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2647 00000100:00080000:0.0:1788203375.489321:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203375.489326:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203375.489332:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203375.489340:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203375.489350:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442966400/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203375.489417:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203375/real 1788203375] req@ffff9023450b9500 x1875064442966016/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203375.489442:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442966016/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203375.489456:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203375.489461:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203375.489467:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442966016:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203375.489507:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442966144:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203375.489554:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203375.489570:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203375/real 1788203375] req@ffff9023450b8380 x1875064442966144/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203375.489584:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442966144/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203375.489595:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203375.489598:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203375.489602:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442966144:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203375.489610:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442966272:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203375.489633:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203375.489648:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203375/real 1788203375] req@ffff9023450b8a80 x1875064442966272/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203375.489662:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442966272/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203375.489677:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203375.489681:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203375.489685:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442966272:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203375.489693:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442966400:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203375.489718:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203375.489730:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203375/real 1788203375] req@ffff9023450bb800 x1875064442966400/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203375.489743:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442966400/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203375.489752:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203375.489757:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203375.489760:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442966400:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203376.320980:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203377.345022:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203378.369088:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203379.393087:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203380.417098:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203380.609031:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203380.609047:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203380.609052:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203380.609067:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2652 00000100:00080000:2.0:1788203380.609082:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203380.609091:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203380.609146:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442966528/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203380.609184:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203380.609192:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203380.609194:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203380.609199:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2652 00000100:00080000:2.0:1788203380.609204:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203380.609209:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203380.609222:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203380.609237:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442966656/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203380.609247:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203380.609251:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203380.609254:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203380.609259:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2652 00000100:00080000:2.0:1788203380.609263:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203380.609267:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203380.609274:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203380.609280:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442966528:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203380.609292:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442966784/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203380.609307:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203380.609313:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203380.609315:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203380.609320:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2652 00000100:00080000:2.0:1788203380.609324:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203380.609328:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203380.609334:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203380.609347:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442966912/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203380.609371:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203380.609419:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203380/real 1788203380] req@ffff90234598dc00 x1875064442966528/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203380.609441:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442966528/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203380.609450:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203380.609453:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203380.609456:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442966528:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203380.609463:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442966656:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203380.609483:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203380.609492:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203380/real 1788203380] req@ffff90234598f800 x1875064442966656/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203380.609500:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442966656/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203380.609506:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203380.609508:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203380.609510:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442966656:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203380.609530:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442966784:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203380.609542:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203380.609548:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203380/real 1788203380] req@ffff90234598ce00 x1875064442966784/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203380.609555:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442966784/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203380.609561:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203380.609564:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203380.609566:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442966784:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203380.609569:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442966912:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203380.609578:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203380.609584:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203380/real 1788203380] req@ffff90234598c700 x1875064442966912/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203380.609592:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442966912/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203380.609597:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203380.609599:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203380.609601:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442966912:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203381.441035:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203382.465103:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203383.489028:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203384.513086:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203385.537139:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203385.728986:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203385.728997:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203385.729002:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203385.729012:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2657 00000100:00080000:0.0:1788203385.729025:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203385.729033:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203385.729077:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442967040/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203385.729102:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203385.729105:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203385.729106:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203385.729109:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2657 00000100:00080000:0.0:1788203385.729112:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203385.729114:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203385.729123:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203385.729133:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442967168/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203385.729141:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203385.729143:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203385.729145:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203385.729147:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2657 00000100:00080000:0.0:1788203385.729149:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203385.729152:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203385.729155:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203385.729159:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442967040:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203385.729165:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442967296/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203385.729173:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203385.729176:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203385.729177:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203385.729180:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2657 00000100:00080000:0.0:1788203385.729182:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203385.729184:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203385.729187:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203385.729195:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442967424/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203385.729239:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203385.729291:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203385/real 1788203385] req@ffff9023450bad80 x1875064442967040/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203385.729309:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442967040/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203385.729321:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203385.729325:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203385.729330:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442967040:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203385.729338:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442967168:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203385.729364:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203385.729377:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203385/real 1788203385] req@ffff9023450b9c00 x1875064442967168/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203385.729386:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442967168/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203385.729406:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203385.729409:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203385.729411:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442967168:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203385.729435:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442967296:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203385.729451:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203385.729458:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203385/real 1788203385] req@ffff9023450b9f80 x1875064442967296/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203385.729466:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442967296/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203385.729472:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203385.729475:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203385.729478:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442967296:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203385.729485:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442967424:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203385.729504:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203385.729516:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203385/real 1788203385] req@ffff9023450b8700 x1875064442967424/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203385.729529:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442967424/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203385.729537:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203385.729540:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203385.729543:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442967424:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203386.561084:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203387.585107:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203388.609042:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203389.633042:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203390.657137:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203390.849029:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203390.849042:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203390.849047:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203390.849059:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2662 00000100:00080000:2.0:1788203390.849072:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203390.849090:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203390.849145:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064442967552/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203390.849177:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203390.849181:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203390.849184:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203390.849189:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2662 00000100:00080000:2.0:1788203390.849192:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203390.849196:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203390.849207:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203390.849221:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064442967680/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203390.849225:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442967552:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203390.849233:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203390.849237:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203390.849238:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203390.849241:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2662 00000100:00080000:2.0:1788203390.849244:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203390.849247:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203390.849251:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203390.849259:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442967808/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203390.849266:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203390.849269:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203390.849270:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203390.849273:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2662 00000100:00080000:2.0:1788203390.849275:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203390.849277:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203390.849280:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203390.849291:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442967936/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203390.849317:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203390.849366:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203390/real 1788203390] req@ffff90234598ca80 x1875064442967552/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203390.849382:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064442967552/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203390.849391:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203390.849395:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203390.849398:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442967552:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203390.849422:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442967680:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203390.849438:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203390.849447:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203390/real 1788203390] req@ffff90234598e680 x1875064442967680/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203390.849455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064442967680/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203390.849461:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203390.849463:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203390.849466:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442967680:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203390.849470:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442967808:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203390.849481:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203390.849488:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203390/real 1788203390] req@ffff90234598d880 x1875064442967808/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203390.849496:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442967808/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203390.849502:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203390.849503:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203390.849505:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442967808:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203390.849509:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442967936:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203390.849552:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203390.849561:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203390/real 1788203390] req@ffff90234598c700 x1875064442967936/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203390.849569:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442967936/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203390.849575:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203390.849577:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203390.849579:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442967936:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203391.681053:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203392.705083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203393.729051:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203394.753059:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203395.777085:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203395.969062:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203395.969077:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203395.969083:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203395.969097:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2667 00000100:00080000:0.0:1788203395.969114:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203395.969123:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203395.969179:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442968064/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203395.969215:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203395.969219:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203395.969221:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203395.969227:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2667 00000100:00080000:0.0:1788203395.969232:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203395.969238:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203395.969251:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203395.969267:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442968192/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203395.969278:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442968064:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203395.969282:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203395.969290:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203395.969293:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203395.969297:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2667 00000100:00080000:0.0:1788203395.969301:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203395.969305:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203395.969312:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203395.969330:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442968320/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203395.969341:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203395.969345:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203395.969347:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203395.969350:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2667 00000100:00080000:0.0:1788203395.969355:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203395.969360:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203395.969367:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203395.969373:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203395.969386:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442968448/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203395.969442:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203395/real 1788203395] req@ffff9023450baa00 x1875064442968064/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203395.969467:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442968064/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203395.969480:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203395.969485:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203395.969490:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442968064:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203395.969520:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442968192:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203395.969546:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203395.969561:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203395/real 1788203395] req@ffff9023450b9180 x1875064442968192/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203395.969575:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442968192/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203395.969585:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203395.969588:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203395.969593:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442968192:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203395.969600:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442968320:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203395.969620:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203395.969632:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203395/real 1788203395] req@ffff9023450b8000 x1875064442968320/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203395.969645:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442968320/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203395.969654:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203395.969658:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203395.969662:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442968320:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203395.969668:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442968448:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203395.969689:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203395.969701:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203395/real 1788203395] req@ffff9023450ba300 x1875064442968448/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203395.969713:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442968448/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203395.969723:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203395.969726:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203395.969730:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442968448:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203396.801113:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203397.825095:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203398.849048:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203399.873107:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203400.897070:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203401.089055:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203401.089071:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203401.089076:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203401.089090:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2672 00000100:00080000:2.0:1788203401.089105:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203401.089113:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203401.089162:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442968576/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203401.089191:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203401.089196:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203401.089198:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203401.089201:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2672 00000100:00080000:2.0:1788203401.089204:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203401.089207:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203401.089217:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203401.089227:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442968704/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203401.089235:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203401.089238:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203401.089239:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203401.089242:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2672 00000100:00080000:2.0:1788203401.089244:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203401.089246:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203401.089250:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203401.089253:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442968576:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203401.089262:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442968832/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203401.089269:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203401.089272:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203401.089273:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203401.089275:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2672 00000100:00080000:2.0:1788203401.089277:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203401.089280:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203401.089284:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203401.089291:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442968960/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203401.089356:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203401.089419:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203401/real 1788203401] req@ffff90234598ce00 x1875064442968576/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203401.089448:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442968576/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203401.089461:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203401.089466:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203401.089470:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442968576:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203401.089478:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442968704:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203401.089502:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203401.089512:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203401/real 1788203401] req@ffff90234598f800 x1875064442968704/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203401.089522:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442968704/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203401.089528:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203401.089530:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203401.089533:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442968704:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203401.089559:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442968832:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203401.089581:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203401.089588:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203401/real 1788203401] req@ffff90234598dc00 x1875064442968832/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203401.089597:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442968832/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203401.089603:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203401.089605:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203401.089607:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442968832:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203401.089611:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442968960:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203401.089620:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203401.089626:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203401/real 1788203401] req@ffff90234598f480 x1875064442968960/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203401.089634:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442968960/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203401.089644:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203401.089646:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203401.089648:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442968960:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203401.921052:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203402.945080:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203403.969104:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203404.993067:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203406.017109:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203406.208977:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203406.208988:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203406.208993:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203406.209004:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2677 00000100:00080000:2.0:1788203406.209018:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203406.209027:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203406.209077:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442969088/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203406.209106:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203406.209109:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203406.209111:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203406.209118:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2677 00000100:00080000:2.0:1788203406.209122:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203406.209125:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203406.209135:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203406.209147:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442969216/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203406.209156:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203406.209158:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203406.209160:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203406.209163:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2677 00000100:00080000:2.0:1788203406.209166:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203406.209171:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203406.209171:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442969088:192.168.201.130@tcp:250:kworker.0 00000008:00100000:2.0:1788203406.209178:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203406.209187:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442969344/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203406.209193:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203406.209196:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203406.209197:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203406.209200:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2677 00000100:00080000:2.0:1788203406.209201:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203406.209204:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203406.209207:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203406.209215:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442969472/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203406.209277:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203406.209325:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203406/real 1788203406] req@ffff90234598c380 x1875064442969088/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203406.209343:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442969088/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203406.209352:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203406.209356:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203406.209359:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442969088:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203406.209392:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442969216:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203406.209418:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203406.209432:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203406/real 1788203406] req@ffff90234598d180 x1875064442969216/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203406.209446:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442969216/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203406.209452:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203406.209454:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203406.209457:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442969216:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203406.209462:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442969344:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203406.209479:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203406.209487:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203406/real 1788203406] req@ffff90234598e300 x1875064442969344/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203406.209495:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442969344/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203406.209501:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203406.209503:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203406.209504:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442969344:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203406.209508:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442969472:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203406.209519:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203406.209524:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203406/real 1788203406] req@ffff90234598df80 x1875064442969472/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203406.209532:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442969472/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203406.209537:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203406.209538:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203406.209540:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442969472:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203407.041116:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203408.065203:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203409.089071:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203410.113088:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203411.137093:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203411.329037:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203411.329056:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203411.329062:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203411.329079:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2682 00000100:00080000:2.0:1788203411.329096:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203411.329108:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203411.329178:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442969600/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203411.329217:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203411.329225:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203411.329228:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203411.329235:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2682 00000100:00080000:2.0:1788203411.329241:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203411.329246:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203411.329253:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442969600:192.168.201.130@tcp:250:kworker.0 00000008:00100000:2.0:1788203411.329263:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203411.329284:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442969728/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203411.329299:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203411.329303:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203411.329306:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203411.329312:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2682 00000100:00080000:2.0:1788203411.329327:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203411.329332:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:0.0:1788203411.329339:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:2.0:1788203411.329340:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203411.329357:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442969856/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203411.329379:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203411.329383:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203411.329385:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203411.329390:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2682 00000100:00080000:2.0:1788203411.329395:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203411.329400:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203411.329406:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203411/real 1788203411] req@ffff90234598fb80 x1875064442969600/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788203411.329409:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203411.329430:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442969984/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203411.329434:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442969600/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203411.329460:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203411.329466:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203411.329472:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442969600:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203411.329504:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442969728:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203411.329534:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203411.329549:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203411/real 1788203411] req@ffff90234598ed80 x1875064442969728/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203411.329558:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442969728/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203411.329565:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203411.329567:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203411.329570:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442969728:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203411.329574:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442969856:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203411.329593:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203411.329600:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203411/real 1788203411] req@ffff90234598f100 x1875064442969856/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203411.329608:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442969856/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203411.329615:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203411.329617:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203411.329619:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442969856:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203411.329622:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442969984:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203411.329633:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203411.329640:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203411/real 1788203411] req@ffff90234598df80 x1875064442969984/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203411.329648:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442969984/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203411.329654:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203411.329655:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203411.329657:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442969984:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203412.161099:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203413.185041:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203414.209134:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203415.233200:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203416.257034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203416.449029:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203416.449038:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203416.449042:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203416.449051:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2687 00000100:00080000:0.0:1788203416.449087:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203416.449094:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203416.449131:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442970112/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203416.449157:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203416.449160:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203416.449161:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203416.449164:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2687 00000100:00080000:0.0:1788203416.449166:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203416.449169:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203416.449176:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203416.449186:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442970240/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203416.449202:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442970112:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203416.449203:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203416.449210:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203416.449213:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203416.449216:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2687 00000100:00080000:0.0:1788203416.449218:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203416.449221:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203416.449225:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203416.449233:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442970368/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203416.449240:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203416.449243:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203416.449245:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203416.449247:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2687 00000100:00080000:0.0:1788203416.449250:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203416.449253:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203416.449255:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203416.449262:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442970496/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203416.449276:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203416.449321:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203416/real 1788203416] req@ffff9023450ba300 x1875064442970112/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203416.449336:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442970112/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203416.449345:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203416.449348:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203416.449351:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442970112:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203416.449370:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442970240:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203416.449387:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203416.449396:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203416/real 1788203416] req@ffff9023450b8000 x1875064442970240/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203416.449404:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442970240/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203416.449410:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203416.449412:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203416.449414:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442970240:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203416.449417:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442970368:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203416.449428:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203416.449433:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203416/real 1788203416] req@ffff9023450b9180 x1875064442970368/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203416.449440:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442970368/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203416.449450:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203416.449452:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203416.449454:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442970368:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203416.449457:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442970496:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203416.449468:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203416.449477:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203416/real 1788203416] req@ffff9023450baa00 x1875064442970496/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203416.449488:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442970496/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203416.449497:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203416.449500:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203416.449504:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442970496:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203417.281162:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203418.305072:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203419.329075:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203420.353041:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203421.377072:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203421.569056:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203421.569065:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203421.569069:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203421.569078:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2693 00000100:00080000:1.0:1788203421.569087:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203421.569092:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203421.569132:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442970624/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203421.569153:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203421.569156:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203421.569157:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203421.569160:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2693 00000100:00080000:1.0:1788203421.569162:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203421.569165:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203421.569172:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203421.569181:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442970752/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203421.569194:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788203421.569199:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442970624:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203421.569206:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203421.569209:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203421.569212:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2693 00000100:00080000:1.0:1788203421.569214:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203421.569217:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203421.569219:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203421.569229:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442970880/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203421.569236:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203421.569238:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203421.569244:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203421.569247:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2693 00000100:00080000:1.0:1788203421.569249:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203421.569252:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203421.569255:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203421.569263:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442971008/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203421.569274:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203421.569337:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203421/real 1788203421] req@ffff90234519ea00 x1875064442970624/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203421.569354:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442970624/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203421.569363:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203421.569368:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203421.569373:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442970624:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203421.569405:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442970752:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203421.569432:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203421.569445:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203421/real 1788203421] req@ffff90234519c700 x1875064442970752/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203421.569459:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442970752/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203421.569465:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203421.569468:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203421.569470:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442970752:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203421.569475:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442970880:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203421.569490:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203421.569497:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203421/real 1788203421] req@ffff90234519fb80 x1875064442970880/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203421.569505:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442970880/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203421.569510:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203421.569512:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203421.569514:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442970880:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203421.569517:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442971008:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203421.569537:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203421.569547:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203421/real 1788203421] req@ffff90234519df80 x1875064442971008/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203421.569555:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442971008/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203421.569560:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203421.569562:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203421.569565:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442971008:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203422.401082:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203423.425011:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203424.449060:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203425.473185:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203426.497082:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203426.689023:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203426.689038:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203426.689041:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203426.689050:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2698 00000100:00080000:0.0:1788203426.689060:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203426.689066:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203426.689102:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442971136/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203426.689128:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203426.689131:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203426.689132:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203426.689135:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2698 00000100:00080000:0.0:1788203426.689138:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203426.689140:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203426.689149:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203426.689166:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442971264/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203426.689181:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442971136:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203426.689201:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203426.689205:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203426.689207:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203426.689212:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2698 00000100:00080000:0.0:1788203426.689217:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203426.689222:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203426.689227:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203426.689243:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442971392/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203426.689254:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788203426.689259:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788203426.689260:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203426.689264:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203426.689269:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2698 00000100:00080000:0.0:1788203426.689273:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203426.689277:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203426.689281:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203426.689292:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442971520/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203426.689309:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203426/real 1788203426] req@ffff9023450b8700 x1875064442971136/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203426.689333:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442971136/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203426.689344:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203426.689347:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203426.689350:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442971136:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203426.689369:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442971264:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203426.689388:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203426.689398:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203426/real 1788203426] req@ffff9023450b9f80 x1875064442971264/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203426.689407:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442971264/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203426.689423:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203426.689427:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203426.689431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442971264:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203426.689438:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442971392:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203426.689456:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203426.689464:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203426/real 1788203426] req@ffff9023450b9c00 x1875064442971392/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203426.689472:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442971392/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203426.689477:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203426.689480:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203426.689482:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442971392:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203426.689485:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442971520:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203426.689495:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203426.689501:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203426/real 1788203426] req@ffff9023450bad80 x1875064442971520/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203426.689508:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442971520/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203426.689514:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203426.689515:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203426.689518:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442971520:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203427.521044:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203428.545036:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203429.569014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203430.593103:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203431.617010:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203431.808962:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203431.808973:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203431.808977:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203431.808987:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2703 00000100:00080000:2.0:1788203431.808999:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203431.809006:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203431.809049:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442971648/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203431.809071:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203431.809074:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203431.809075:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203431.809078:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2703 00000100:00080000:2.0:1788203431.809081:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203431.809084:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203431.809091:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203431.809099:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442971776/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203431.809106:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203431.809109:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203431.809110:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203431.809112:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2703 00000100:00080000:2.0:1788203431.809115:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203431.809117:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203431.809120:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203431.809131:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442971904/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203431.809138:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203431.809142:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203431.809143:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203431.809143:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442971648:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203431.809148:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2703 00000100:00080000:2.0:1788203431.809152:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203431.809156:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203431.809158:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203431.809166:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442972032/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203431.809213:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203431.809255:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203431/real 1788203431] req@ffff90234598e300 x1875064442971648/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203431.809271:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442971648/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203431.809279:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203431.809283:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203431.809286:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442971648:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203431.809292:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442971776:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203431.809307:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203431.809316:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203431/real 1788203431] req@ffff90234598d180 x1875064442971776/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203431.809323:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442971776/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203431.809329:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203431.809330:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203431.809332:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442971776:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203431.809363:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442971904:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203431.809376:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203431.809384:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203431/real 1788203431] req@ffff90234598c380 x1875064442971904/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203431.809392:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442971904/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203431.809398:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203431.809399:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203431.809402:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442971904:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203431.809405:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442972032:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203431.809415:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203431.809421:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203431/real 1788203431] req@ffff90234598f480 x1875064442972032/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203431.809428:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442972032/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203431.809432:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203431.809434:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203431.809436:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442972032:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203432.641055:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203433.665014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203434.689041:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203435.713023:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203436.737077:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203436.928933:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203436.928942:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203436.928945:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203436.928953:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2708 00000100:00080000:2.0:1788203436.928962:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203436.928968:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203436.928999:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442972160/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203436.929022:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203436.929026:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203436.929028:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203436.929031:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2708 00000100:00080000:2.0:1788203436.929033:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203436.929035:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203436.929043:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203436.929051:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442972288/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203436.929058:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203436.929060:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203436.929062:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203436.929065:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2708 00000100:00080000:2.0:1788203436.929067:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203436.929069:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203436.929072:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203436.929079:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442972416/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203436.929085:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203436.929087:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203436.929089:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203436.929094:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2708 00000100:00100000:3.0:1788203436.929095:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442972160:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203436.929096:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203436.929102:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203436.929108:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203436.929116:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442972544/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203436.929199:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442972288:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203436.929241:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442972416:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203436.929272:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442972544:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203437.762072:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203438.785039:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203439.809022:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788203440.001276:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:0.0:1788203440.001323:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788203440.001340:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788203440.001386:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788203440.001404:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788203440.001419:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:1.0:1788203440.001452:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203436/real 1788203440] req@ffff90234598ce00 x1875064442972416/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203440.001487:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442972416/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203440.001504:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203440.001511:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203440.001516:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442972416:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203440.001528:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203436/real 1788203440] req@ffff90234598c700 x1875064442972544/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203440.001543:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442972544/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203440.001553:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203440.001558:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203440.001562:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442972544:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203440.001570:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203436/real 1788203440] req@ffff90234598dc00 x1875064442972160/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203440.001584:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442972160/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203440.001592:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203440.001596:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203440.001600:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442972160:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203440.001607:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203436/real 1788203440] req@ffff90234598f800 x1875064442972288/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203440.001619:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442972288/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203440.001627:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203440.001630:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203440.001634:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442972288:192.168.201.130@tcp:38:kworker.0 00010000:00010000:1.0:1788203440.833110:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203441.857025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203442.049052:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203442.049061:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203442.049064:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203442.049072:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2713 00000100:00080000:2.0:1788203442.049082:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203442.049088:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203442.049123:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442972672/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203442.049159:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203442.049163:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203442.049165:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203442.049171:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2713 00000100:00080000:2.0:1788203442.049175:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203442.049180:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203442.049191:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203442.049207:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064442972800/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203442.049217:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203442.049220:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203442.049222:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203442.049227:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2713 00000100:00080000:2.0:1788203442.049231:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203442.049235:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203442.049240:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203442.049252:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064442972928/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203442.049262:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203442.049267:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203442.049269:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203442.049274:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2713 00000100:00080000:2.0:1788203442.049279:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203442.049283:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203442.049290:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203442.049304:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442973056/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203442.049363:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442972672:192.168.201.130@tcp:250:kworker.0 00000800:00000100:1.0:1788203442.049514:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203442.049589:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203442/real 1788203442] req@ffff90234598d880 x1875064442972672/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203442.049616:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442972672/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203442.049671:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203442.049677:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203442.049683:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442972672:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203442.049693:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442972800:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203442.049730:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203442.049746:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203442/real 1788203442] req@ffff90234598e680 x1875064442972800/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203442.049760:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064442972800/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203442.049776:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203442.049780:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203442.049783:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442972800:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203442.049790:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442972928:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203442.049812:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203442.049824:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203442/real 1788203442] req@ffff90234598ca80 x1875064442972928/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203442.049838:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064442972928/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203442.049855:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203442.049859:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203442.049862:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442972928:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203442.049870:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442973056:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203442.049942:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203442.049957:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203442/real 1788203442] req@ffff90234598f800 x1875064442973056/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203442.049969:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442973056/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203442.049977:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203442.049980:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203442.049984:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442973056:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203442.881047:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203443.905096:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203444.929037:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203445.953010:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203446.977100:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203447.168999:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203447.169010:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203447.169014:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203447.169027:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2718 00000100:00080000:2.0:1788203447.169040:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203447.169049:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203447.169093:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442973184/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203447.169116:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203447.169122:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203447.169123:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203447.169127:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2718 00000100:00080000:2.0:1788203447.169129:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203447.169132:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203447.169142:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203447.169162:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442973312/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203447.169173:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788203447.169176:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442973184:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203447.169177:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203447.169180:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203447.169196:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2718 00000100:00080000:2.0:1788203447.169201:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203447.169205:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203447.169209:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203447.169225:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442973440/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203447.169233:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203447.169238:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203447.169240:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203447.169244:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2718 00000100:00080000:2.0:1788203447.169247:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203447.169251:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203447.169258:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788203447.169267:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203447.169274:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442973568/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203447.169330:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203447/real 1788203447] req@ffff90234598dc00 x1875064442973184/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203447.169345:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442973184/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203447.169354:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203447.169358:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203447.169364:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442973184:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203447.169387:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442973312:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203447.169403:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203447.169413:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203447/real 1788203447] req@ffff90234598c700 x1875064442973312/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203447.169421:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442973312/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203447.169426:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203447.169429:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203447.169431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442973312:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203447.169435:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442973440:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203447.169446:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203447.169451:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203447/real 1788203447] req@ffff90234598ce00 x1875064442973440/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203447.169459:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442973440/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203447.169465:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203447.169467:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203447.169469:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442973440:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203447.169472:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442973568:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203447.169484:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203447.169490:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203447/real 1788203447] req@ffff90234598f480 x1875064442973568/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203447.169498:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442973568/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203447.169503:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203447.169505:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203447.169506:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442973568:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203448.001014:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203449.025027:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203450.049055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203451.073035:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203452.097016:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203452.289027:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203452.289039:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203452.289044:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203452.289054:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2723 00000100:00080000:0.0:1788203452.289064:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203452.289071:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203452.289104:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442973696/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203452.289128:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203452.289131:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203452.289133:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203452.289135:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2723 00000100:00080000:0.0:1788203452.289138:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203452.289140:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203452.289148:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203452.289157:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442973824/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203452.289166:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788203452.289169:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442973696:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203452.289170:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203452.289174:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203452.289183:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2723 00000100:00080000:0.0:1788203452.289186:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203452.289191:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203452.289196:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203452.289211:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442973952/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203452.289222:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203452.289226:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203452.289228:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788203452.289233:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788203452.289236:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2723 00000100:00080000:0.0:1788203452.289242:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203452.289246:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203452.289251:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203452.289262:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442974080/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203452.289279:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203452/real 1788203452] req@ffff9023450bb800 x1875064442973696/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203452.289296:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442973696/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203452.289305:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203452.289308:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203452.289311:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442973696:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203452.289328:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442973824:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203452.289344:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203452.289352:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203452/real 1788203452] req@ffff9023450b8a80 x1875064442973824/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203452.289361:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442973824/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203452.289367:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203452.289369:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203452.289371:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442973824:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203452.289374:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442973952:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203452.289385:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203452.289390:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203452/real 1788203452] req@ffff9023450b8380 x1875064442973952/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203452.289398:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442973952/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203452.289403:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203452.289405:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203452.289407:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442973952:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203452.289410:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442974080:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203452.289419:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203452.289425:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203452/real 1788203452] req@ffff9023450b9500 x1875064442974080/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203452.289432:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442974080/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203452.289442:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203452.289459:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203452.289463:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442974080:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203453.121013:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203454.145044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203455.169019:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203456.193096:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203457.217036:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203457.409003:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203457.409013:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203457.409018:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203457.409030:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2728 00000100:00080000:0.0:1788203457.409043:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203457.409050:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203457.409101:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442974208/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203457.409134:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203457.409138:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203457.409140:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203457.409143:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2728 00000100:00080000:0.0:1788203457.409146:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203457.409149:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203457.409158:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203457.409167:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442974336/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203457.409175:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203457.409177:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203457.409179:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203457.409181:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2728 00000100:00080000:0.0:1788203457.409184:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203457.409187:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203457.409190:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203457.409196:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442974464/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203457.409202:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203457.409205:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203457.409207:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203457.409210:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2728 00000100:00080000:0.0:1788203457.409212:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203457.409215:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203457.409218:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203457.409228:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442974208:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203457.409229:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442974592/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203457.409337:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203457.409419:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203457/real 1788203457] req@ffff9023450bb480 x1875064442974208/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203457.409443:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442974208/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203457.409455:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203457.409460:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203457.409464:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442974208:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203457.409473:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442974336:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203457.409500:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203457.409515:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203457/real 1788203457] req@ffff9023450ba680 x1875064442974336/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203457.409528:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442974336/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203457.409544:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203457.409548:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203457.409551:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442974336:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203457.409558:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442974464:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203457.409579:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203457.409591:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203457/real 1788203457] req@ffff9023450b8e00 x1875064442974464/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203457.409604:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442974464/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203457.409622:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203457.409626:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203457.409630:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442974464:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203457.409659:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442974592:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203457.409691:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203457.409705:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203457/real 1788203457] req@ffff9023450bbb80 x1875064442974592/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203457.409719:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442974592/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203457.409728:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203457.409732:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203457.409736:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442974592:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203458.241092:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203459.265078:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203460.289096:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203461.313063:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203462.337177:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203462.529011:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203462.529029:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203462.529034:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203462.529047:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2734 00000100:00080000:0.0:1788203462.529060:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203462.529068:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203462.529121:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442974720/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203462.529157:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203462.529160:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203462.529162:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203462.529168:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2734 00000100:00080000:0.0:1788203462.529171:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203462.529174:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203462.529184:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203462.529194:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442974848/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203462.529202:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203462.529204:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203462.529206:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203462.529209:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2734 00000100:00080000:0.0:1788203462.529211:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203462.529215:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203462.529219:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203462.529228:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442974976/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203462.529235:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203462.529238:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203462.529240:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203462.529242:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2734 00000100:00080000:0.0:1788203462.529244:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203462.529246:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203462.529249:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203462.529255:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442975104/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203462.529283:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442974720:192.168.201.130@tcp:250:kworker.0 00000800:00000100:1.0:1788203462.529359:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203462.529411:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203462/real 1788203462] req@ffff9023450bbb80 x1875064442974720/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203462.529429:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442974720/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203462.529440:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203462.529445:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203462.529449:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442974720:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203462.529458:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442974848:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203462.529480:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203462.529492:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203462/real 1788203462] req@ffff9023450b8e00 x1875064442974848/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203462.529515:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442974848/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203462.529523:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203462.529527:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203462.529531:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442974848:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203462.529537:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442974976:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203462.529567:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203462.529578:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203462/real 1788203462] req@ffff9023450ba680 x1875064442974976/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203462.529591:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442974976/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203462.529600:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203462.529603:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203462.529616:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442974976:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203462.529642:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442975104:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203462.529662:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203462.529673:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203462/real 1788203462] req@ffff9023450bb480 x1875064442975104/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203462.529686:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442975104/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203462.529694:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203462.529697:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203462.529701:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442975104:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203463.361100:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203464.385055:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203465.409045:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203466.433038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203467.457019:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203467.649027:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203467.649038:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203467.649043:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203467.649052:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2739 00000100:00080000:2.0:1788203467.649062:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203467.649067:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203467.649100:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442975232/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203467.649125:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203467.649128:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203467.649129:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203467.649132:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2739 00000100:00080000:2.0:1788203467.649135:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203467.649137:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203467.649144:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203467.649154:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442975360/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203467.649162:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203467.649165:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203467.649166:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203467.649172:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2739 00000100:00100000:1.0:1788203467.649174:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442975232:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203467.649176:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203467.649203:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203467.649208:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203467.649220:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442975488/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203467.649228:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203467.649230:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203467.649231:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203467.649234:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2739 00000100:00080000:2.0:1788203467.649239:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203467.649241:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203467.649244:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788203467.649248:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203467.649255:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442975616/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203467.649285:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203467/real 1788203467] req@ffff90234598c380 x1875064442975232/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203467.649298:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442975232/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203467.649306:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203467.649309:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203467.649312:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442975232:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203467.649331:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442975360:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203467.649345:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203467.649353:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203467/real 1788203467] req@ffff90234598d180 x1875064442975360/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203467.649361:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442975360/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203467.649366:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203467.649368:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203467.649370:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442975360:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203467.649374:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442975488:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203467.649386:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203467.649392:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203467/real 1788203467] req@ffff90234598e300 x1875064442975488/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203467.649399:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442975488/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203467.649405:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203467.649406:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203467.649408:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442975488:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203467.649411:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442975616:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203467.649420:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203467.649425:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203467/real 1788203467] req@ffff90234598df80 x1875064442975616/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203467.649432:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442975616/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203467.649437:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203467.649439:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203467.649441:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442975616:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203468.481049:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203469.505025:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203470.529057:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203471.553077:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203472.577038:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203472.768989:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203472.769001:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203472.769006:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203472.769017:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2744 00000100:00080000:2.0:1788203472.769029:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203472.769037:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203472.769075:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442975744/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203472.769121:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203472.769126:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203472.769128:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203472.769131:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2744 00000100:00080000:2.0:1788203472.769134:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203472.769137:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203472.769145:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203472.769155:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442975872/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203472.769163:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203472.769165:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203472.769166:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203472.769168:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2744 00000100:00080000:2.0:1788203472.769170:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203472.769173:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203472.769176:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203472.769181:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442976000/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203472.769188:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203472.769190:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203472.769192:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203472.769195:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2744 00000100:00080000:2.0:1788203472.769197:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203472.769200:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203472.769203:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203472.769215:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442975744:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203472.769215:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442976128/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203472.769280:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203472.769314:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203472/real 1788203472] req@ffff90234598f100 x1875064442975744/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203472.769356:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442975744/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203472.769369:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203472.769374:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203472.769379:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442975744:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203472.769385:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442975872:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203472.769405:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203472.769419:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203472/real 1788203472] req@ffff90234598ed80 x1875064442975872/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203472.769432:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442975872/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203472.769441:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203472.769445:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203472.769450:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442975872:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203472.769458:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442976000:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203472.769477:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203472.769485:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203472/real 1788203472] req@ffff90234598fb80 x1875064442976000/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203472.769498:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442976000/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203472.769504:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203472.769506:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203472.769508:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442976000:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203472.769538:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442976128:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203472.769558:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203472.769569:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203472/real 1788203472] req@ffff90234598df80 x1875064442976128/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203472.769581:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442976128/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203472.769589:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203472.769592:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203472.769595:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442976128:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203473.601120:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203474.625017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203475.649069:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203476.673003:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203477.697106:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203477.888988:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203477.888998:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203477.889011:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203477.889021:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2749 00000100:00080000:0.0:1788203477.889031:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203477.889036:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203477.889074:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442976256/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203477.889098:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203477.889101:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203477.889103:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203477.889106:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2749 00000100:00080000:0.0:1788203477.889108:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203477.889110:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203477.889119:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203477.889130:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442976384/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203477.889138:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203477.889141:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203477.889144:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203477.889147:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2749 00000100:00100000:1.0:1788203477.889153:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442976256:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203477.889156:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203477.889162:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203477.889166:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203477.889175:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442976512/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203477.889181:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203477.889183:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203477.889185:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203477.889187:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2749 00000100:00080000:0.0:1788203477.889189:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203477.889192:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203477.889195:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203477.889203:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442976640/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203477.889229:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203477.889290:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203477/real 1788203477] req@ffff9023450b9500 x1875064442976256/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203477.889312:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442976256/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203477.889326:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203477.889331:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203477.889336:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442976256:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203477.889364:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442976384:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203477.889403:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203477.889419:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203477/real 1788203477] req@ffff9023450b8380 x1875064442976384/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203477.889434:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442976384/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203477.889444:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203477.889447:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203477.889451:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442976384:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203477.889458:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442976512:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203477.889477:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203477.889486:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203477/real 1788203477] req@ffff9023450b8a80 x1875064442976512/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203477.889495:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442976512/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203477.889506:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203477.889509:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203477.889511:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442976512:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203477.889515:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442976640:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203477.889526:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203477.889533:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203477/real 1788203477] req@ffff9023450bb800 x1875064442976640/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203477.889540:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442976640/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203477.889546:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203477.889548:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203477.889549:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442976640:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203478.721100:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203479.745061:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203480.769073:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203481.793122:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203482.817093:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203483.008981:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203483.008995:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203483.009000:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203483.009014:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2754 00000100:00080000:0.0:1788203483.009029:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203483.009038:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203483.009086:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442976768/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203483.009116:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203483.009120:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203483.009123:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203483.009128:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2754 00000100:00080000:0.0:1788203483.009132:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203483.009138:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203483.009149:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203483.009155:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442976768:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203483.009165:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442976896/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203483.009176:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203483.009179:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203483.009182:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203483.009186:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2754 00000100:00080000:0.0:1788203483.009191:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203483.009196:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203483.009200:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203483.009212:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442977024/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203483.009222:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203483.009226:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203483.009228:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203483.009232:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2754 00000100:00080000:0.0:1788203483.009237:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788203483.009238:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788203483.009241:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203483.009246:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203483.009258:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442977152/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203483.009285:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203483/real 1788203483] req@ffff9023450bad80 x1875064442976768/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203483.009307:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442976768/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203483.009319:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203483.009323:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203483.009326:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442976768:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203483.009351:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442976896:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203483.009371:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203483.009381:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203483/real 1788203483] req@ffff9023450b9c00 x1875064442976896/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203483.009391:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442976896/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203483.009397:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203483.009400:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203483.009402:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442976896:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203483.009407:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442977024:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203483.009421:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203483.009430:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203483/real 1788203483] req@ffff9023450b9f80 x1875064442977024/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203483.009439:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442977024/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203483.009444:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203483.009446:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203483.009448:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442977024:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203483.009451:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442977152:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203483.009462:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203483.009468:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203483/real 1788203483] req@ffff9023450b8700 x1875064442977152/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203483.009475:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442977152/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203483.009480:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203483.009481:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203483.009483:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442977152:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203483.841075:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203484.865070:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203485.889060:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203486.913031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203487.937079:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203488.128988:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203488.128999:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203488.129003:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203488.129014:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2759 00000100:00080000:0.0:1788203488.129025:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203488.129033:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203488.129069:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442977280/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203488.129092:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203488.129099:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203488.129101:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203488.129105:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2759 00000100:00080000:0.0:1788203488.129109:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203488.129114:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203488.129123:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203488.129129:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442977280:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203488.129136:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442977408/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203488.129145:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203488.129149:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203488.129150:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203488.129154:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2759 00000100:00080000:0.0:1788203488.129156:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203488.129159:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203488.129163:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203488.129172:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442977536/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203488.129180:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203488.129183:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203488.129186:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788203488.129186:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788203488.129189:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2759 00000100:00080000:0.0:1788203488.129191:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203488.129194:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203488.129199:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203488.129206:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442977664/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203488.129224:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203488/real 1788203488] req@ffff9023450baa00 x1875064442977280/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203488.129248:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442977280/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203488.129259:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203488.129263:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203488.129267:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442977280:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203488.129290:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442977408:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203488.129309:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203488.129318:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203488/real 1788203488] req@ffff9023450b9180 x1875064442977408/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203488.129326:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442977408/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203488.129332:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203488.129334:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203488.129336:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442977408:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203488.129340:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442977536:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203488.129350:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203488.129356:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203488/real 1788203488] req@ffff9023450b8000 x1875064442977536/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203488.129363:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442977536/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203488.129372:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203488.129375:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203488.129377:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442977536:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203488.129380:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442977664:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203488.129389:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203488.129394:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203488/real 1788203488] req@ffff9023450ba300 x1875064442977664/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203488.129401:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442977664/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203488.129406:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203488.129408:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203488.129410:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442977664:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203488.961068:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203489.984998:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203491.009028:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203492.033046:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203493.057048:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203493.249000:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203493.249008:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203493.249012:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203493.249022:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2764 00000100:00080000:1.0:1788203493.249033:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203493.249040:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203493.249081:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442977792/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203493.249111:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203493.249115:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203493.249117:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203493.249121:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2764 00000100:00080000:1.0:1788203493.249125:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203493.249130:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203493.249140:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203493.249149:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442977792:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203493.249157:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442977920/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203493.249170:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203493.249174:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203493.249175:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203493.249179:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2764 00000100:00080000:1.0:1788203493.249184:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203493.249188:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203493.249195:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203493.249209:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442978048/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203493.249219:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788203493.249221:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203493.249225:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203493.249227:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203493.249232:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2764 00000100:00080000:1.0:1788203493.249237:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203493.249240:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203493.249244:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203493.249254:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442978176/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203493.249262:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203493/real 1788203493] req@ffff90234519c000 x1875064442977792/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203493.249280:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442977792/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203493.249288:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203493.249291:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203493.249294:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442977792:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203493.249313:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442977920:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203493.249340:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203493.249349:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203493/real 1788203493] req@ffff90234519e300 x1875064442977920/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203493.249358:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442977920/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203493.249364:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203493.249367:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203493.249369:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442977920:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203493.249373:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442978048:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203493.249388:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203493.249398:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203493/real 1788203493] req@ffff90234519f800 x1875064442978048/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203493.249411:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442978048/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203493.249421:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203493.249425:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203493.249429:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442978048:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203493.249437:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442978176:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203493.249457:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203493.249469:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203493/real 1788203493] req@ffff90234519ed80 x1875064442978176/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203493.249479:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442978176/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203493.249485:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203493.249487:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203493.249489:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442978176:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203494.081099:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203495.105106:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203496.129078:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203497.153028:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203498.177050:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203498.369014:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203498.369026:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203498.369030:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203498.369041:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2769 00000100:00080000:0.0:1788203498.369052:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203498.369059:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203498.369100:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442978304/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203498.369128:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203498.369131:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203498.369132:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203498.369136:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2769 00000100:00080000:0.0:1788203498.369139:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203498.369142:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203498.369149:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203498.369161:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442978432/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203498.369169:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203498.369171:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203498.369173:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203498.369176:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2769 00000100:00080000:0.0:1788203498.369179:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203498.369184:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442978304:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203498.369185:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203498.369191:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203498.369199:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442978560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203498.369206:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203498.369209:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203498.369210:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203498.369213:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2769 00000100:00080000:0.0:1788203498.369215:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203498.369217:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203498.369220:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203498.369228:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442978688/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203498.369254:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203498.369296:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203498/real 1788203498] req@ffff9023450ba300 x1875064442978304/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203498.369316:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442978304/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203498.369328:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203498.369332:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203498.369336:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442978304:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203498.369345:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442978432:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203498.369369:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203498.369382:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203498/real 1788203498] req@ffff9023450b8000 x1875064442978432/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203498.369411:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442978432/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203498.369427:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203498.369431:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203498.369435:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442978432:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203498.369462:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442978560:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203498.369483:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203498.369496:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203498/real 1788203498] req@ffff9023450b9180 x1875064442978560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203498.369509:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442978560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203498.369519:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203498.369522:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203498.369525:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442978560:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203498.369532:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442978688:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203498.369552:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203498.369564:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203498/real 1788203498] req@ffff9023450baa00 x1875064442978688/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203498.369577:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442978688/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203498.369585:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203498.369588:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203498.369592:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442978688:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203499.201063:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203500.225038:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203501.248990:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203502.273046:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203503.297105:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203503.489020:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203503.489031:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203503.489035:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203503.489047:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2775 00000100:00080000:1.0:1788203503.489059:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203503.489066:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203503.489121:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442978816/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203503.489153:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203503.489156:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203503.489157:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203503.489160:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2775 00000100:00080000:1.0:1788203503.489162:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203503.489165:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203503.489175:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203503.489188:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442978944/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203503.489202:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203503.489205:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203503.489207:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203503.489209:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2775 00000100:00080000:1.0:1788203503.489212:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203503.489216:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203503.489219:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203503.489229:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442979072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203503.489237:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442978816:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203503.489243:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203503.489252:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203503.489253:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203503.489258:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2775 00000100:00080000:1.0:1788203503.489260:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203503.489265:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203503.489270:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203503.489279:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442979200/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203503.489351:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442978944:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203503.489389:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442979072:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203503.489405:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442979200:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203504.321012:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203505.345029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203506.369016:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788203506.561133:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:0.0:1788203506.561171:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788203506.561185:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788203506.561222:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788203506.561233:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788203506.561243:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:2.0:1788203506.561297:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203503/real 1788203506] req@ffff90234519f800 x1875064442979072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203506.561326:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442979072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203506.561339:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203506.561345:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203506.561350:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442979072:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203506.561359:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203503/real 1788203506] req@ffff90234519e300 x1875064442979200/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203506.561372:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442979200/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203506.561380:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203506.561384:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203506.561387:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442979200:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203506.561395:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203503/real 1788203506] req@ffff90234519d880 x1875064442978816/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203506.561408:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442978816/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203506.561417:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203506.561420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203506.561424:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442978816:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203506.561432:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203503/real 1788203506] req@ffff90234519ed80 x1875064442978944/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203506.561444:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442978944/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203506.561454:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203506.561457:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203506.561461:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442978944:192.168.201.130@tcp:38:kworker.0 00010000:00010000:0.0:1788203507.393055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203508.417011:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203508.608994:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203508.609002:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203508.609006:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203508.609014:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2780 00000100:00080000:1.0:1788203508.609023:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203508.609029:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203508.609071:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442979328/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203508.609094:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203508.609097:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203508.609098:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203508.609101:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2780 00000100:00080000:1.0:1788203508.609103:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203508.609106:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203508.609113:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203508.609122:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442979456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203508.609129:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203508.609132:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203508.609134:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203508.609139:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2780 00000100:00080000:1.0:1788203508.609142:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203508.609146:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203508.609149:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203508.609155:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442979584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203508.609162:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203508.609164:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203508.609165:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203508.609168:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2780 00000100:00080000:1.0:1788203508.609170:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203508.609178:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203508.609181:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442979328:192.168.201.130@tcp:250:kworker.0 00000008:00100000:1.0:1788203508.609184:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203508.609207:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442979712/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203508.609296:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203508.609356:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203508/real 1788203508] req@ffff90234519c000 x1875064442979328/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203508.609379:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442979328/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203508.609399:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203508.609404:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203508.609409:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442979328:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203508.609418:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442979456:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203508.609445:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203508.609461:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203508/real 1788203508] req@ffff90234519df80 x1875064442979456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203508.609486:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442979456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203508.609495:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203508.609499:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203508.609503:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442979456:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203508.609544:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442979584:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203508.609570:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203508.609584:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203508/real 1788203508] req@ffff90234519fb80 x1875064442979584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203508.609597:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442979584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203508.609611:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203508.609615:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203508.609619:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442979584:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203508.609628:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442979712:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203508.609651:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203508.609663:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203508/real 1788203508] req@ffff90234519c700 x1875064442979712/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203508.609675:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442979712/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203508.609683:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203508.609687:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203508.609690:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442979712:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203509.441014:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203510.465099:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203511.489055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203512.513060:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203513.537075:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203513.729064:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203513.729078:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203513.729083:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203513.729094:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2785 00000100:00080000:2.0:1788203513.729107:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203513.729115:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203513.729158:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442979840/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203513.729188:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203513.729191:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203513.729192:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203513.729196:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2785 00000100:00080000:2.0:1788203513.729199:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203513.729204:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203513.729215:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203513.729224:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442979968/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203513.729230:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442979840:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203513.729235:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203513.729239:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203513.729240:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203513.729244:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2785 00000100:00080000:2.0:1788203513.729246:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203513.729249:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203513.729261:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203513.729276:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442980096/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203513.729286:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203513.729290:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203513.729293:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203513.729296:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2785 00000100:00080000:2.0:1788203513.729301:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788203513.729302:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788203513.729311:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203513.729317:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203513.729332:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442980224/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203513.729362:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203513/real 1788203513] req@ffff90234598e300 x1875064442979840/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203513.729406:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442979840/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203513.729418:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203513.729423:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203513.729428:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442979840:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203513.729453:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442979968:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203513.729478:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203513.729488:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203513/real 1788203513] req@ffff90234598d180 x1875064442979968/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203513.729498:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442979968/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203513.729504:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203513.729506:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203513.729509:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442979968:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203513.729513:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442980096:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203513.729524:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203513.729530:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203513/real 1788203513] req@ffff90234598c380 x1875064442980096/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203513.729538:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442980096/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203513.729548:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203513.729550:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203513.729553:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442980096:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203513.729556:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442980224:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203513.729565:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203513.729571:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203513/real 1788203513] req@ffff90234598f480 x1875064442980224/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203513.729578:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442980224/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203513.729584:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203513.729586:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203513.729588:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442980224:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203514.561113:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203515.585039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203516.609090:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203517.633035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203518.657093:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203518.848971:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203518.848980:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203518.848984:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203518.848993:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2790 00000100:00080000:2.0:1788203518.849004:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203518.849012:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203518.849050:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442980352/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203518.849083:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203518.849088:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203518.849090:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203518.849095:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2790 00000100:00080000:2.0:1788203518.849100:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203518.849105:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203518.849115:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203518.849132:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442980480/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203518.849143:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442980352:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203518.849148:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203518.849154:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203518.849156:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203518.849161:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2790 00000100:00080000:2.0:1788203518.849165:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203518.849169:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203518.849174:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203518.849187:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442980608/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203518.849195:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203518.849198:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203518.849199:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203518.849203:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2790 00000100:00080000:2.0:1788203518.849206:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203518.849209:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203518.849213:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788203518.849217:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203518.849228:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442980736/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203518.849263:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203518/real 1788203518] req@ffff90234598ce00 x1875064442980352/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203518.849285:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442980352/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203518.849315:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203518.849320:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203518.849326:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442980352:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203518.849352:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442980480:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203518.849375:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203518.849387:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203518/real 1788203518] req@ffff90234598c700 x1875064442980480/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203518.849400:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442980480/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203518.849408:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203518.849412:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203518.849415:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442980480:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203518.849422:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442980608:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203518.849439:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203518.849449:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203518/real 1788203518] req@ffff90234598dc00 x1875064442980608/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203518.849460:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442980608/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203518.849468:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203518.849472:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203518.849475:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442980608:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203518.849480:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442980736:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203518.849497:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203518.849507:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203518/real 1788203518] req@ffff90234598f800 x1875064442980736/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203518.849518:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442980736/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203518.849526:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203518.849529:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203518.849532:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442980736:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203519.681522:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203520.705022:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203521.729034:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203522.753036:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203523.777003:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203523.968972:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203523.968988:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203523.968992:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203523.969001:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2795 00000100:00080000:0.0:1788203523.969011:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203523.969016:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203523.969050:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442980864/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203523.969074:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203523.969076:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203523.969077:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203523.969080:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2795 00000100:00080000:0.0:1788203523.969083:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203523.969086:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203523.969095:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203523.969108:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442980992/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203523.969118:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203523.969122:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203523.969128:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788203523.969128:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442980864:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203523.969135:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2795 00000100:00080000:0.0:1788203523.969139:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203523.969143:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203523.969148:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203523.969159:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442981120/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203523.969168:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203523.969172:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203523.969174:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203523.969178:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2795 00000100:00080000:0.0:1788203523.969181:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203523.969187:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:1.0:1788203523.969188:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:0.0:1788203523.969194:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203523.969219:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442981248/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203523.969250:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203523/real 1788203523] req@ffff9023450b8700 x1875064442980864/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203523.969268:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442980864/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203523.969284:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203523.969288:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203523.969292:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442980864:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203523.969311:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442980992:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203523.969329:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203523.969348:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203523/real 1788203523] req@ffff9023450b9f80 x1875064442980992/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203523.969359:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442980992/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203523.969367:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203523.969370:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203523.969373:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442980992:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203523.969378:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442981120:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203523.969394:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203523.969402:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203523/real 1788203523] req@ffff9023450b9c00 x1875064442981120/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203523.969412:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442981120/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203523.969420:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203523.969423:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203523.969426:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442981120:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203523.969432:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442981248:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203523.969445:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203523.969453:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203523/real 1788203523] req@ffff9023450bad80 x1875064442981248/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203523.969463:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442981248/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203523.969470:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203523.969473:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203523.969476:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442981248:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203524.801035:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203525.825004:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203526.849086:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203527.873006:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203528.897019:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203529.088965:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203529.088974:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203529.088978:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203529.088987:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2800 00000100:00080000:0.0:1788203529.088998:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203529.089004:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203529.089043:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442981376/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203529.089063:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203529.089066:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203529.089067:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203529.089069:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2800 00000100:00080000:0.0:1788203529.089072:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203529.089075:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203529.089082:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203529.089089:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442981504/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203529.089096:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203529.089099:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203529.089100:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203529.089103:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2800 00000100:00080000:0.0:1788203529.089105:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788203529.089127:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442981376:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203529.089138:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203529.089145:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203529.089158:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442981632/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203529.089167:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203529.089170:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203529.089171:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203529.089174:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2800 00000100:00080000:0.0:1788203529.089198:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203529.089203:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203529.089208:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788203529.089215:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203529.089221:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442981760/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203529.089275:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203529/real 1788203529] req@ffff9023450bb800 x1875064442981376/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203529.089295:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442981376/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203529.089307:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203529.089311:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203529.089315:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442981376:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203529.089323:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442981504:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203529.089341:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203529.089350:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203529/real 1788203529] req@ffff9023450b8a80 x1875064442981504/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203529.089359:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442981504/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203529.089366:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203529.089369:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203529.089372:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442981504:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203529.089394:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442981632:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203529.089413:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203529.089420:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203529/real 1788203529] req@ffff9023450b8380 x1875064442981632/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203529.089428:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442981632/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203529.089433:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203529.089435:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203529.089437:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442981632:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203529.089440:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442981760:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203529.089449:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203529.089455:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203529/real 1788203529] req@ffff9023450b9500 x1875064442981760/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203529.089465:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442981760/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203529.089470:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203529.089471:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203529.089473:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442981760:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203529.921034:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203530.945039:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203531.969053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203532.993028:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203534.017115:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203534.208982:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203534.208990:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203534.208993:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203534.209003:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2805 00000100:00080000:0.0:1788203534.209012:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203534.209017:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203534.209069:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442981888/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203534.209093:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203534.209096:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203534.209098:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203534.209101:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2805 00000100:00080000:0.0:1788203534.209104:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203534.209108:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203534.209115:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203534.209132:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442981888:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203534.209139:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442982016/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203534.209149:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203534.209151:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203534.209153:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203534.209156:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2805 00000100:00080000:0.0:1788203534.209160:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203534.209163:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203534.209167:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203534.209176:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442982144/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203534.209182:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203534.209185:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203534.209186:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203534.209189:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2805 00000100:00080000:0.0:1788203534.209192:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203534.209194:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203534.209196:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788203534.209203:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203534.209205:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442982272/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203534.209242:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203534/real 1788203534] req@ffff9023450bb480 x1875064442981888/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203534.209256:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442981888/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203534.209264:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203534.209267:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203534.209270:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442981888:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203534.209288:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442982016:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203534.209304:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203534.209312:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203534/real 1788203534] req@ffff9023450ba680 x1875064442982016/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203534.209320:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442982016/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203534.209325:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203534.209327:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203534.209329:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442982016:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203534.209333:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442982144:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203534.209353:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203534.209360:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203534/real 1788203534] req@ffff9023450b8e00 x1875064442982144/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203534.209368:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442982144/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203534.209374:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203534.209376:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203534.209378:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442982144:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203534.209381:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442982272:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203534.209391:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203534.209397:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203534/real 1788203534] req@ffff9023450bbb80 x1875064442982272/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203534.209404:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442982272/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203534.209413:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203534.209415:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203534.209417:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442982272:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203535.041049:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203536.065067:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203537.089024:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203538.113043:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203539.137029:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203539.328976:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203539.328986:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203539.328990:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203539.328998:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2810 00000100:00080000:1.0:1788203539.329008:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203539.329013:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203539.329045:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442982400/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203539.329072:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203539.329075:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203539.329077:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203539.329080:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2810 00000100:00080000:1.0:1788203539.329082:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203539.329085:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203539.329093:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203539.329102:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442982528/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203539.329109:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203539.329112:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203539.329114:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788203539.329118:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442982400:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203539.329119:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2810 00000100:00080000:1.0:1788203539.329123:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203539.329130:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203539.329134:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203539.329147:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442982656/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203539.329154:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203539.329156:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203539.329157:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203539.329159:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2810 00000100:00080000:1.0:1788203539.329161:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203539.329163:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203539.329166:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203539.329173:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442982784/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203539.329191:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203539.329226:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203539/real 1788203539] req@ffff90234519ea00 x1875064442982400/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203539.329240:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442982400/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203539.329248:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203539.329251:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203539.329254:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442982400:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203539.329271:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442982528:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203539.329286:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203539.329294:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203539/real 1788203539] req@ffff90234519c380 x1875064442982528/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203539.329302:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442982528/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203539.329308:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203539.329310:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203539.329312:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442982528:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203539.329316:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442982656:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203539.329327:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203539.329333:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203539/real 1788203539] req@ffff90234519f480 x1875064442982656/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203539.329340:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442982656/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203539.329346:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203539.329348:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203539.329349:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442982656:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203539.329353:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442982784:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203539.329362:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203539.329368:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203539/real 1788203539] req@ffff90234519dc00 x1875064442982784/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203539.329374:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442982784/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203539.329379:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203539.329381:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203539.329383:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442982784:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203540.161071:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203541.185083:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203542.209084:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203543.233017:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203544.257041:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203544.449074:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203544.449094:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203544.449098:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203544.449106:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2815 00000100:00080000:0.0:1788203544.449116:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203544.449121:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203544.449157:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442982912/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203544.449189:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203544.449193:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203544.449195:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203544.449199:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2815 00000100:00080000:0.0:1788203544.449203:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203544.449208:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203544.449219:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203544.449233:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442983040/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203544.449243:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203544.449249:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203544.449252:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788203544.449252:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442982912:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203544.449265:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2815 00000100:00080000:0.0:1788203544.449270:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203544.449274:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203544.449279:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203544.449297:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442983168/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203544.449308:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203544.449313:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203544.449316:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203544.449319:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2815 00000100:00080000:0.0:1788203544.449325:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203544.449331:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203544.449337:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788203544.449345:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203544.449354:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442983296/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203544.449394:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203544/real 1788203544] req@ffff9023450bbb80 x1875064442982912/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203544.449415:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442982912/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203544.449428:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203544.449433:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203544.449438:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442982912:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203544.449480:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442983040:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203544.449502:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203544.449512:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203544/real 1788203544] req@ffff9023450b8e00 x1875064442983040/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203544.449522:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442983040/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203544.449528:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203544.449531:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203544.449533:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442983040:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203544.449537:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442983168:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203544.449550:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203544.449557:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203544/real 1788203544] req@ffff9023450ba680 x1875064442983168/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203544.449569:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442983168/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203544.449585:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203544.449588:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203544.449592:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442983168:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203544.449599:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442983296:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203544.449619:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203544.449628:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203544/real 1788203544] req@ffff9023450bb480 x1875064442983296/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203544.449636:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442983296/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203544.449642:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203544.449644:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203544.449646:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442983296:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203545.281048:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203546.305052:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203547.329042:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203548.353104:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203549.377029:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203549.569050:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203549.569057:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203549.569061:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203549.569069:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2821 00000100:00080000:2.0:1788203549.569079:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203549.569084:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203549.569136:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064442983424/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203549.569171:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203549.569175:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203549.569177:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203549.569180:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2821 00000100:00080000:2.0:1788203549.569183:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203549.569186:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203549.569195:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203549.569210:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064442983552/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203549.569220:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203549.569222:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203549.569223:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203549.569233:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2821 00000100:00100000:1.0:1788203549.569237:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442983424:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203549.569238:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203549.569250:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203549.569256:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203549.569267:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064442983680/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203549.569274:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203549.569277:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203549.569279:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203549.569282:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2821 00000100:00080000:2.0:1788203549.569284:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203549.569288:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203549.569292:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203549.569302:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442983808/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203549.569329:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203549.569390:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203549/real 1788203549] req@ffff90234598ca80 x1875064442983424/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203549.569407:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064442983424/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203549.569416:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203549.569420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203549.569424:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442983424:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203549.569450:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442983552:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788203549.569467:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203549.569476:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203549/real 1788203549] req@ffff90234598e680 x1875064442983552/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203549.569484:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064442983552/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203549.569490:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203549.569493:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203549.569495:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442983552:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788203549.569499:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442983680:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203549.569511:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203549.569518:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203549/real 1788203549] req@ffff90234598d880 x1875064442983680/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203549.569526:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064442983680/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203549.569532:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203549.569534:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203549.569536:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442983680:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788203549.569539:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442983808:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788203549.569551:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203549.569557:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203549/real 1788203549] req@ffff90234598f800 x1875064442983808/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203549.569565:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442983808/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203549.569570:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203549.569572:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788203549.569574:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442983808:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203550.401042:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203551.425021:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203552.449010:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203553.473047:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203554.497069:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203554.689042:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203554.689056:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203554.689062:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203554.689077:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2826 00000100:00080000:0.0:1788203554.689092:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203554.689102:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203554.689151:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442983936/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203554.689187:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203554.689191:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203554.689194:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203554.689201:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2826 00000100:00080000:0.0:1788203554.689207:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203554.689217:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203554.689237:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203554.689243:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442983936:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203554.689260:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442984064/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203554.689277:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203554.689282:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203554.689285:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203554.689297:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2826 00000100:00080000:0.0:1788203554.689303:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203554.689309:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203554.689317:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788203554.689324:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203554.689335:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442984192/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203554.689349:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203554.689354:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203554.689356:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203554.689364:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2826 00000100:00080000:0.0:1788203554.689369:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203554.689376:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203554.689382:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203554/real 1788203554] req@ffff9023450b9500 x1875064442983936/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788203554.689385:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203554.689409:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442984320/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203554.689422:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442983936/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203554.689437:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203554.689443:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203554.689448:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442983936:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203554.689477:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442984064:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203554.689505:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203554.689520:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203554/real 1788203554] req@ffff9023450b8380 x1875064442984064/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203554.689533:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442984064/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203554.689542:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203554.689546:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203554.689550:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442984064:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203554.689557:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442984192:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203554.689583:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203554.689598:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203554/real 1788203554] req@ffff9023450b8a80 x1875064442984192/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203554.689615:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442984192/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203554.689625:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203554.689629:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203554.689631:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442984192:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203554.689640:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442984320:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203554.689663:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203554.689699:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203554/real 1788203554] req@ffff9023450bb800 x1875064442984320/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203554.689718:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442984320/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203554.689732:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203554.689735:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203554.689740:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442984320:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203555.521035:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203556.545034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203557.569032:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203558.593070:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203559.617067:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203559.808969:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203559.808981:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203559.808986:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203559.808995:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2831 00000100:00080000:0.0:1788203559.809005:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203559.809012:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203559.809050:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442984448/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203559.809079:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203559.809083:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203559.809085:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203559.809089:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2831 00000100:00080000:0.0:1788203559.809093:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203559.809098:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203559.809107:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203559.809109:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442984448:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203559.809125:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442984576/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203559.809138:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203559.809142:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203559.809144:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203559.809148:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2831 00000100:00080000:0.0:1788203559.809152:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203559.809156:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203559.809161:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203559.809174:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442984704/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203559.809180:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788203559.809192:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203559.809196:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203559.809198:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203559.809203:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2831 00000100:00080000:0.0:1788203559.809207:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203559.809212:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203559.809217:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203559.809223:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203559/real 1788203559] req@ffff9023450bad80 x1875064442984448/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203559.809231:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442984832/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203559.809260:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442984448/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203559.809270:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203559.809274:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203559.809280:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442984448:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203559.809306:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442984576:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203559.809332:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203559.809344:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203559/real 1788203559] req@ffff9023450b9c00 x1875064442984576/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203559.809354:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442984576/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203559.809359:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203559.809362:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203559.809364:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442984576:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203559.809368:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442984704:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203559.809381:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203559.809388:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203559/real 1788203559] req@ffff9023450b9f80 x1875064442984704/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203559.809395:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442984704/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203559.809406:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203559.809408:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203559.809410:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442984704:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203559.809413:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442984832:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203559.809423:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203559.809428:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203559/real 1788203559] req@ffff9023450b8700 x1875064442984832/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203559.809436:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442984832/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203559.809441:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203559.809443:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203559.809444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442984832:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203560.641040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203561.665064:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203562.689033:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203563.713030:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203564.737079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203564.929051:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203564.929071:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203564.929076:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203564.929087:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2836 00000100:00080000:0.0:1788203564.929102:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203564.929109:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203564.929157:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442984960/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203564.929188:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203564.929193:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203564.929195:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203564.929200:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2836 00000100:00080000:0.0:1788203564.929204:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203564.929209:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203564.929222:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203564.929240:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442985088/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203564.929254:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203564.929257:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203564.929260:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203564.929266:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2836 00000100:00080000:0.0:1788203564.929275:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203564.929276:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442984960:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203564.929286:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203564.929296:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203564.929327:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442985216/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203564.929337:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203564.929341:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203564.929343:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203564.929347:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2836 00000800:00000100:2.0:1788203564.929353:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788203564.929354:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203564.929366:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203564.929373:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203564.929385:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442985344/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203564.929405:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203564/real 1788203564] req@ffff9023450baa00 x1875064442984960/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203564.929428:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442984960/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203564.929438:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203564.929442:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203564.929445:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442984960:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203564.929452:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442985088:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203564.929470:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203564.929483:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203564/real 1788203564] req@ffff9023450b9180 x1875064442985088/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203564.929535:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442985088/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203564.929546:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203564.929549:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203564.929551:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442985088:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203564.929573:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442985216:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203564.929590:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203564.929599:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203564/real 1788203564] req@ffff9023450b8000 x1875064442985216/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203564.929607:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442985216/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203564.929613:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203564.929615:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203564.929617:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442985216:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203564.929621:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442985344:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203564.929635:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203564.929656:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203564/real 1788203564] req@ffff9023450ba300 x1875064442985344/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203564.929669:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442985344/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203564.929678:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203564.929681:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203564.929685:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442985344:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203565.761108:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203566.785055:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203567.809075:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203568.833091:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203569.857083:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203570.049003:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203570.049011:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203570.049015:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203570.049023:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2841 00000100:00080000:1.0:1788203570.049033:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203570.049038:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203570.049073:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442985472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203570.049094:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203570.049097:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203570.049098:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203570.049101:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2841 00000100:00080000:1.0:1788203570.049103:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203570.049106:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203570.049113:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203570.049123:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442985600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203570.049131:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203570.049133:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203570.049134:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203570.049139:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2841 00000100:00080000:1.0:1788203570.049143:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203570.049143:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442985472:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203570.049147:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203570.049150:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203570.049156:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442985728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203570.049161:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203570.049164:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203570.049165:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203570.049167:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2841 00000100:00080000:1.0:1788203570.049169:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203570.049171:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203570.049173:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203570.049178:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442985856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203570.049277:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442985600:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203570.049303:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442985728:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203570.049319:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442985856:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203570.881030:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203571.904995:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203572.929027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788203573.121198:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788203573.121241:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788203573.121255:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788203573.121298:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788203573.121312:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788203573.121324:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788203573.121382:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203570/real 1788203573] req@ffff90234519dc00 x1875064442985600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203573.121431:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442985600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203573.121452:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203573.121459:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203573.121464:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442985600:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203573.121475:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203570/real 1788203573] req@ffff90234519f480 x1875064442985728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203573.121490:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442985728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203573.121500:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203573.121504:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203573.121509:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442985728:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203573.121516:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203570/real 1788203573] req@ffff90234519c380 x1875064442985856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203573.121530:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442985856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203573.121539:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203573.121543:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203573.121546:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442985856:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203573.121556:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203570/real 1788203573] req@ffff90234519f100 x1875064442985472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203573.121571:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442985472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203573.121580:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203573.121583:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203573.121587:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442985472:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788203573.952991:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203574.977056:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203575.170297:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203575.170308:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203575.170326:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203575.170340:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2846 00000100:00080000:0.0:1788203575.170352:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203575.170361:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203575.170404:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442985984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203575.170440:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203575.170445:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203575.170447:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203575.170452:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2846 00000100:00080000:0.0:1788203575.170456:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203575.170461:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203575.170472:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203575.170490:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442986112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203575.170500:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203575.170505:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203575.170507:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203575.170509:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2846 00000100:00080000:0.0:1788203575.170513:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203575.170526:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203575.170532:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203575.170547:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442985984:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203575.170555:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442986240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203575.170580:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203575.170585:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203575.170587:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203575.170592:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2846 00000100:00080000:0.0:1788203575.170599:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203575.170605:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203575.170612:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203575.170633:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442986368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203575.170719:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203575.170797:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203575/real 1788203575] req@ffff9023450ba300 x1875064442985984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203575.170827:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442985984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203575.170842:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203575.170848:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203575.170853:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442985984:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203575.170864:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442986112:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203575.170962:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203575.170978:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203575/real 1788203575] req@ffff9023450b8000 x1875064442986112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203575.170992:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442986112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203575.171003:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203575.171008:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203575.171012:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442986112:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203575.171050:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442986240:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203575.171075:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203575.171088:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203575/real 1788203575] req@ffff9023450b9180 x1875064442986240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203575.171100:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442986240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203575.171110:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203575.171115:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203575.171118:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442986240:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203575.171126:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442986368:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203575.171152:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203575.171350:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203575/real 1788203575] req@ffff9023450baa00 x1875064442986368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203575.171364:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442986368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203575.171371:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203575.171375:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203575.171379:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442986368:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203576.001010:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203577.025057:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203578.049027:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203579.073107:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203580.097085:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203580.288996:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203580.289007:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203580.289011:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203580.289031:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2851 00000100:00080000:2.0:1788203580.289045:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203580.289054:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203580.289106:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442986496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203580.289138:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203580.289141:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203580.289142:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203580.289147:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2851 00000100:00080000:2.0:1788203580.289151:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203580.289155:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203580.289167:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203580.289184:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442986624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203580.289195:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203580.289200:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203580.289203:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203580.289207:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2851 00000100:00080000:2.0:1788203580.289211:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203580.289216:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203580.289237:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442986496:192.168.201.130@tcp:250:kworker.0 00000008:00100000:2.0:1788203580.289241:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203580.289264:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442986752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203580.289281:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203580.289287:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203580.289290:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203580.289295:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2851 00000100:00080000:2.0:1788203580.289315:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203580.289326:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:3.0:1788203580.289328:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:2.0:1788203580.289337:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203580.289359:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442986880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203580.289385:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203580/real 1788203580] req@ffff90234598dc00 x1875064442986496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203580.289406:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442986496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203580.289417:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203580.289422:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203580.289426:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442986496:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203580.289436:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442986624:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203580.289459:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203580.289471:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203580/real 1788203580] req@ffff90234598c700 x1875064442986624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203580.289484:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442986624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203580.289493:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203580.289496:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203580.289499:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442986624:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203580.289525:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442986752:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203580.289543:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203580.289554:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203580/real 1788203580] req@ffff90234598ce00 x1875064442986752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203580.289566:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442986752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203580.289575:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203580.289578:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203580.289582:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442986752:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203580.289588:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442986880:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203580.289605:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203580.289625:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203580/real 1788203580] req@ffff90234598f480 x1875064442986880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203580.289641:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442986880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203580.289649:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203580.289653:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203580.289656:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442986880:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203581.121052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203582.145057:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203583.169039:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203584.193025:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203585.217073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203585.408984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203585.408995:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203585.408999:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203585.409009:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2856 00000100:00080000:1.0:1788203585.409021:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203585.409028:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203585.409068:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442987008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203585.409098:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203585.409101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203585.409103:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203585.409107:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2856 00000100:00080000:1.0:1788203585.409110:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203585.409115:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203585.409124:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203585.409161:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442987008:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203585.409162:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442987136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203585.409179:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203585.409183:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203585.409185:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203585.409189:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2856 00000100:00080000:1.0:1788203585.409194:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203585.409197:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203585.409203:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203585.409216:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442987264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203585.409222:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788203585.409232:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203585.409238:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203585.409240:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203585.409244:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2856 00000100:00080000:1.0:1788203585.409248:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203585.409252:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203585.409257:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203585.409269:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442987392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203585.409277:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203585/real 1788203585] req@ffff90234519ea00 x1875064442987008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203585.409343:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442987008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203585.409351:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203585.409354:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203585.409357:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442987008:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203585.409376:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442987136:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203585.409393:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203585.409402:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203585/real 1788203585] req@ffff90234519c700 x1875064442987136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203585.409410:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442987136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203585.409416:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203585.409418:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203585.409420:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442987136:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203585.409424:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442987264:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203585.409434:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203585.409441:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203585/real 1788203585] req@ffff90234519fb80 x1875064442987264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203585.409448:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442987264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203585.409453:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203585.409455:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203585.409457:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442987264:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203585.409461:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442987392:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203585.409470:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203585.409476:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203585/real 1788203585] req@ffff90234519df80 x1875064442987392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203585.409483:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442987392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203585.409488:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203585.409489:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203585.409491:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442987392:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203586.241010:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203587.265076:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203588.289064:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203589.313022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203590.337025:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203590.528993:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203590.529001:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203590.529005:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203590.529013:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2862 00000100:00080000:0.0:1788203590.529023:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203590.529029:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203590.529104:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442987520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203590.529128:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203590.529132:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203590.529134:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203590.529138:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2862 00000100:00080000:0.0:1788203590.529142:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203590.529147:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203590.529167:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203590.529179:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442987520:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203590.529184:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442987648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203590.529199:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203590.529203:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203590.529205:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203590.529210:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2862 00000100:00080000:0.0:1788203590.529215:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203590.529219:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203590.529224:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203590.529234:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442987776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203590.529241:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203590.529244:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203590.529246:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203590.529248:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2862 00000100:00080000:0.0:1788203590.529251:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203590.529255:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203590.529261:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203590.529261:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203590.529272:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442987904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203590.529310:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203590/real 1788203590] req@ffff9023450b8700 x1875064442987520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203590.529332:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442987520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203590.529344:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203590.529349:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203590.529355:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442987520:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203590.529392:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442987648:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203590.529422:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203590.529441:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203590/real 1788203590] req@ffff9023450b9f80 x1875064442987648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203590.529457:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442987648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203590.529467:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203590.529472:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203590.529477:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442987648:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203590.529486:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442987776:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203590.529510:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203590.529525:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203590/real 1788203590] req@ffff9023450b9c00 x1875064442987776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203590.529540:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442987776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203590.529551:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203590.529555:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203590.529560:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442987776:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203590.529568:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442987904:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203590.529659:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203590.529672:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203590/real 1788203590] req@ffff9023450bad80 x1875064442987904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203590.529685:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442987904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203590.529700:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203590.529704:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203590.529708:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442987904:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203591.361013:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203592.384984:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203593.409018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203594.433050:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203595.456987:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203595.649004:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203595.649013:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203595.649017:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203595.649025:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2867 00000100:00080000:0.0:1788203595.649037:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203595.649043:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203595.649079:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442988032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203595.649108:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203595.649112:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203595.649114:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203595.649118:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2867 00000100:00080000:0.0:1788203595.649121:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203595.649124:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203595.649134:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203595.649151:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442988032:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203595.649156:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442988160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203595.649168:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203595.649172:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203595.649173:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203595.649177:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2867 00000100:00080000:0.0:1788203595.649181:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203595.649193:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203595.649198:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203595.649203:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203595.649212:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442988288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203595.649221:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203595.649224:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203595.649226:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203595.649230:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2867 00000100:00080000:0.0:1788203595.649234:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788203595.649235:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203595/real 1788203595] req@ffff9023450bb800 x1875064442988032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203595.649241:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203595.649249:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203595.649253:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442988032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203595.649262:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442988416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203595.649262:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203595.649269:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203595.649284:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442988032:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203595.649307:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442988160:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203595.649323:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203595.649331:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203595/real 1788203595] req@ffff9023450b8a80 x1875064442988160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203595.649338:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442988160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203595.649343:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203595.649345:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203595.649347:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442988160:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203595.649351:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442988288:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203595.649360:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203595.649366:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203595/real 1788203595] req@ffff9023450b8380 x1875064442988288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203595.649372:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442988288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203595.649377:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203595.649378:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203595.649380:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442988288:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203595.649383:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442988416:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203595.649391:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203595.649396:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203595/real 1788203595] req@ffff9023450b9500 x1875064442988416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203595.649403:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442988416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203595.649407:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203595.649408:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203595.649410:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442988416:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203596.481107:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203597.505107:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203598.529083:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203599.553068:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203600.577050:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203600.769035:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203600.769051:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203600.769057:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203600.769069:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2872 00000100:00080000:0.0:1788203600.769082:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203600.769089:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203600.769138:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442988544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203600.769172:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203600.769176:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203600.769178:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203600.769182:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2872 00000100:00080000:0.0:1788203600.769186:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203600.769190:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203600.769202:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203600.769217:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442988672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203600.769222:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442988544:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203600.769229:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203600.769233:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203600.769234:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203600.769238:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2872 00000100:00080000:0.0:1788203600.769241:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203600.769244:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203600.769248:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203600.769256:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442988800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203600.769262:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203600.769265:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203600.769266:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203600.769268:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2872 00000100:00080000:0.0:1788203600.769270:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203600.769272:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203600.769275:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203600.769289:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442988928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203600.769303:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203600.769358:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203600/real 1788203600] req@ffff9023450bb480 x1875064442988544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203600.769379:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442988544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203600.769390:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203600.769395:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203600.769398:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442988544:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203600.769419:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442988672:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203600.769436:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203600.769445:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203600/real 1788203600] req@ffff9023450ba680 x1875064442988672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203600.769453:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442988672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203600.769458:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203600.769461:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203600.769464:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442988672:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203600.769469:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442988800:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203600.769487:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203600.769507:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203600/real 1788203600] req@ffff9023450b8e00 x1875064442988800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203600.769519:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442988800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203600.769532:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203600.769536:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203600.769539:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442988800:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203600.769545:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442988928:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203600.769564:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203600.769575:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203600/real 1788203600] req@ffff9023450bbb80 x1875064442988928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203600.769587:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442988928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203600.769594:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203600.769597:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203600.769600:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442988928:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203601.601019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203602.625080:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203603.649025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203604.673053:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203605.697022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203605.889040:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203605.889050:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203605.889054:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203605.889065:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2877 00000100:00080000:1.0:1788203605.889077:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203605.889084:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203605.889136:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442989056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203605.889164:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203605.889167:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203605.889171:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203605.889174:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2877 00000100:00080000:1.0:1788203605.889177:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203605.889180:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203605.889189:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203605.889199:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442989184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203605.889205:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203605.889208:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203605.889209:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203605.889214:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2877 00000100:00100000:3.0:1788203605.889215:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442989056:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203605.889217:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203605.889222:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203605.889226:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203605.889233:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442989312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203605.889239:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203605.889242:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203605.889243:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203605.889245:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2877 00000100:00080000:1.0:1788203605.889248:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203605.889251:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203605.889254:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203605.889264:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442989440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203605.889286:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203605.889326:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203605/real 1788203605] req@ffff90234519c000 x1875064442989056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203605.889340:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442989056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203605.889349:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203605.889352:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203605.889354:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442989056:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203605.889376:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442989184:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203605.889389:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203605.889397:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203605/real 1788203605] req@ffff90234519ed80 x1875064442989184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203605.889404:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442989184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203605.889409:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203605.889411:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203605.889413:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442989184:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203605.889417:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442989312:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203605.889426:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203605.889431:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203605/real 1788203605] req@ffff90234519d880 x1875064442989312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203605.889438:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442989312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203605.889443:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203605.889444:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203605.889446:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442989312:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203605.889448:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442989440:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203605.889457:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203605.889462:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203605/real 1788203605] req@ffff90234519e300 x1875064442989440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203605.889469:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442989440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203605.889473:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203605.889474:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203605.889476:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442989440:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203606.721071:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203607.745036:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203608.769048:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203609.793118:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203610.817049:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203611.009024:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203611.009036:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203611.009040:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203611.009051:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2882 00000100:00080000:1.0:1788203611.009065:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203611.009073:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203611.009119:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442989568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203611.009149:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203611.009152:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203611.009153:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203611.009156:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2882 00000100:00080000:1.0:1788203611.009159:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203611.009161:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203611.009170:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203611.009181:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442989696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203611.009188:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203611.009191:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203611.009192:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203611.009195:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2882 00000100:00080000:1.0:1788203611.009196:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203611.009199:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203611.009202:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203611.009209:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442989824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203611.009214:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203611.009217:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203611.009218:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203611.009221:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2882 00000100:00080000:1.0:1788203611.009222:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203611.009225:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203611.009227:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203611.009233:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442989952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203611.009296:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442989568:192.168.201.130@tcp:250:kworker.0 00000800:00000100:3.0:1788203611.009437:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203611.009500:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203611/real 1788203611] req@ffff90234519f800 x1875064442989568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203611.009545:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442989568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203611.009554:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203611.009558:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203611.009561:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442989568:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203611.009618:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442989696:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203611.009724:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203611.009752:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203611/real 1788203611] req@ffff90234519e300 x1875064442989696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203611.009788:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442989696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203611.009811:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203611.009822:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203611.009825:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442989696:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203611.009830:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442989824:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203611.009858:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203611.009921:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203611/real 1788203611] req@ffff90234519d880 x1875064442989824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203611.009931:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442989824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203611.009937:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203611.009950:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203611.009953:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442989824:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203611.009957:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442989952:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203611.009984:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203611.010015:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203611/real 1788203611] req@ffff90234519ed80 x1875064442989952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203611.010042:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442989952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203611.010085:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203611.010088:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203611.010091:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442989952:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203611.841047:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203612.865035:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203613.889112:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203614.913056:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203615.937106:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203616.129036:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203616.129051:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203616.129058:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203616.129073:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2887 00000100:00080000:1.0:1788203616.129088:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203616.129098:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203616.129154:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442990080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203616.129204:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203616.129208:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203616.129210:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203616.129216:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2887 00000100:00080000:1.0:1788203616.129220:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203616.129225:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203616.129238:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203616.129253:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442990208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203616.129264:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442990080:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203616.129268:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203616.129276:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203616.129279:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203616.129283:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2887 00000100:00080000:1.0:1788203616.129287:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203616.129292:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203616.129298:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203616.129310:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442990336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203616.129327:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203616.129331:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203616.129333:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203616.129336:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2887 00000100:00080000:1.0:1788203616.129340:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203616.129344:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203616.129349:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203616.129360:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203616.129363:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442990464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203616.129417:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203616/real 1788203616] req@ffff90234519c000 x1875064442990080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203616.129439:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442990080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203616.129450:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203616.129454:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203616.129459:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442990080:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203616.129488:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442990208:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203616.129513:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203616.129529:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203616/real 1788203616] req@ffff90234519df80 x1875064442990208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203616.129542:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442990208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203616.129552:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203616.129555:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203616.129559:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442990208:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203616.129566:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442990336:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203616.129637:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203616.129652:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203616/real 1788203616] req@ffff90234519fb80 x1875064442990336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203616.129667:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442990336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203616.129676:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203616.129680:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203616.129683:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442990336:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203616.129691:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442990464:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203616.129712:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203616.129723:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203616/real 1788203616] req@ffff90234519c700 x1875064442990464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203616.129736:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442990464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203616.129745:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203616.129749:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203616.129753:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442990464:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203616.961062:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203617.985031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203619.009043:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203620.033047:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203621.057081:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203621.248977:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203621.248985:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203621.248988:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203621.248997:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2892 00000100:00080000:1.0:1788203621.249007:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203621.249012:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203621.249045:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442990592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203621.249068:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203621.249071:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203621.249072:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203621.249075:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2892 00000100:00080000:1.0:1788203621.249078:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203621.249080:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203621.249088:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203621.249097:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442990720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203621.249109:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203621.249112:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203621.249113:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203621.249117:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2892 00000100:00080000:1.0:1788203621.249119:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203621.249121:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203621.249124:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203621.249131:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442990848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203621.249141:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442990592:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203621.249142:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203621.249151:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203621.249154:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203621.249159:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2892 00000100:00080000:1.0:1788203621.249163:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203621.249168:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203621.249174:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203621.249188:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442990976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203621.249217:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203621.249259:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203621/real 1788203621] req@ffff90234519ea00 x1875064442990592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203621.249276:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442990592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203621.249286:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203621.249290:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203621.249294:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442990592:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203621.249300:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442990720:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203621.249322:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203621.249335:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203621/real 1788203621] req@ffff90234519f100 x1875064442990720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203621.249348:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442990720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203621.249355:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203621.249357:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203621.249360:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442990720:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203621.249378:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442990848:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203621.249393:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203621.249400:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203621/real 1788203621] req@ffff90234519c380 x1875064442990848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203621.249408:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442990848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203621.249414:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203621.249416:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203621.249418:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442990848:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203621.249422:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442990976:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203621.249431:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203621.249437:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203621/real 1788203621] req@ffff90234519f480 x1875064442990976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203621.249444:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442990976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203621.249449:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203621.249451:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203621.249452:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442990976:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203622.081087:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203623.105021:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203624.129045:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203625.153034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203626.177078:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203626.368986:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203626.369007:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203626.369011:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203626.369022:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2897 00000100:00080000:1.0:1788203626.369035:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203626.369043:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203626.369084:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442991104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203626.369114:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203626.369118:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203626.369119:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203626.369123:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2897 00000100:00080000:1.0:1788203626.369126:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203626.369129:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203626.369139:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203626.369152:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442991232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203626.369161:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203626.369164:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203626.369166:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203626.369168:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2897 00000100:00080000:1.0:1788203626.369172:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203626.369177:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203626.369186:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442991104:192.168.201.130@tcp:250:kworker.0 00000008:00100000:1.0:1788203626.369187:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203626.369203:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442991360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203626.369220:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203626.369223:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203626.369225:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203626.369227:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2897 00000100:00080000:1.0:1788203626.369229:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203626.369232:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203626.369236:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203626.369246:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442991488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203626.369296:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203626.369344:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203626/real 1788203626] req@ffff90234519dc00 x1875064442991104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203626.369362:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442991104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203626.369374:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203626.369382:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203626.369387:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442991104:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203626.369397:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442991232:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203626.369446:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203626.369459:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203626/real 1788203626] req@ffff90234519f480 x1875064442991232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203626.369467:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442991232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203626.369474:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203626.369477:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203626.369479:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442991232:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203626.369501:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442991360:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203626.369522:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203626.369535:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203626/real 1788203626] req@ffff90234519c380 x1875064442991360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203626.369547:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442991360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203626.369555:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203626.369559:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203626.369562:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442991360:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203626.369571:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442991488:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203626.369593:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203626.369609:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203626/real 1788203626] req@ffff90234519f100 x1875064442991488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203626.369623:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442991488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203626.369632:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203626.369635:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203626.369640:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442991488:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203627.201097:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203628.224999:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203629.249065:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203630.273034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203631.297051:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203631.488984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203631.488995:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203631.488999:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203631.489009:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2903 00000100:00080000:1.0:1788203631.489019:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203631.489026:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203631.489061:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442991616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203631.489093:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203631.489097:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203631.489100:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203631.489113:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2903 00000100:00080000:1.0:1788203631.489121:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203631.489126:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203631.489135:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203631.489141:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442991616:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203631.489151:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442991744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203631.489164:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203631.489167:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203631.489169:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203631.489173:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2903 00000100:00080000:1.0:1788203631.489177:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203631.489181:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203631.489187:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203631.489197:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442991872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203631.489206:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788203631.489211:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788203631.489213:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203631.489216:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203631.489221:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2903 00000100:00080000:1.0:1788203631.489226:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203631.489230:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203631.489235:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203631.489247:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442992000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203631.489251:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203631/real 1788203631] req@ffff90234519ea00 x1875064442991616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203631.489282:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442991616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203631.489290:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203631.489293:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203631.489297:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442991616:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203631.489316:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442991744:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203631.489331:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203631.489376:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203631/real 1788203631] req@ffff90234519c700 x1875064442991744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203631.489391:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442991744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203631.489400:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203631.489404:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203631.489407:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442991744:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203631.489414:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442991872:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203631.489433:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203631.489441:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203631/real 1788203631] req@ffff90234519fb80 x1875064442991872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203631.489454:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442991872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203631.489459:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203631.489461:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203631.489464:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442991872:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203631.489467:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442992000:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203631.489477:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203631.489482:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203631/real 1788203631] req@ffff90234519df80 x1875064442992000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203631.489489:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442992000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203631.489494:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203631.489496:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203631.489498:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442992000:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203632.321049:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203633.345085:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203634.369023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203635.393042:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203636.417058:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203636.609215:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203636.609226:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203636.609231:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203636.609240:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2908 00000100:00080000:1.0:1788203636.609267:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203636.609276:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203636.609316:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442992128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203636.609353:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203636.609357:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203636.609359:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203636.609364:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2908 00000100:00080000:1.0:1788203636.609368:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203636.609372:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203636.609381:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203636.609396:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442992256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203636.609408:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203636.609411:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203636.609413:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203636.609418:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2908 00000100:00080000:1.0:1788203636.609421:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203636.609425:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203636.609429:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203636.609438:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442992384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203636.609447:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203636.609450:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203636.609452:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203636.609456:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2908 00000100:00080000:1.0:1788203636.609459:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203636.609462:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203636.609467:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203636.609475:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442992512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203636.609630:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442992128:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203636.609717:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442992256:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203636.609744:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442992384:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203636.609763:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442992512:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203637.441514:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203638.465186:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203639.492367:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788203639.681145:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:1.0:1788203639.681186:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788203639.681216:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788203639.681258:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788203639.681268:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788203639.681275:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:2.0:1788203639.681355:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203636/real 1788203639] req@ffff90234519c000 x1875064442992128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203639.681377:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442992128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203639.681387:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203639.681391:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203639.681395:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442992128:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203639.681402:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203636/real 1788203639] req@ffff90234519ed80 x1875064442992256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203639.681413:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442992256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203639.681420:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203639.681423:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203639.681426:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442992256:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203639.681433:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203636/real 1788203639] req@ffff90234519d880 x1875064442992384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203639.681444:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442992384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203639.681451:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203639.681455:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203639.681458:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442992384:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203639.681465:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203636/real 1788203639] req@ffff90234519e300 x1875064442992512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203639.681476:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442992512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203639.681483:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203639.681487:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203639.681489:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442992512:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203640.513016:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203641.536984:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203641.729035:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203641.729048:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203641.729051:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203641.729060:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2913 00000100:00080000:1.0:1788203641.729069:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203641.729076:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203641.729111:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064442992640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203641.729135:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203641.729139:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203641.729141:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203641.729145:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2913 00000100:00080000:1.0:1788203641.729149:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203641.729153:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203641.729184:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442992640:192.168.201.130@tcp:250:kworker.0 00000008:00100000:1.0:1788203641.729189:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203641.729203:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064442992768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203641.729212:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203641.729215:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203641.729217:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203641.729220:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2913 00000100:00080000:1.0:1788203641.729222:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203641.729225:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203641.729230:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203641.729238:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064442992896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203641.729246:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203641.729249:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203641.729251:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788203641.729253:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788203641.729254:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2913 00000100:00080000:1.0:1788203641.729257:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203641.729260:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203641.729264:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203641.729272:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064442993024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203641.729288:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203641/real 1788203641] req@ffff90234519f800 x1875064442992640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203641.729307:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064442992640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203641.729319:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203641.729322:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203641.729326:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442992640:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203641.729354:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442992768:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203641.729377:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203641.729390:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203641/real 1788203641] req@ffff90234519e300 x1875064442992768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203641.729402:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064442992768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203641.729418:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203641.729421:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203641.729424:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442992768:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203641.729430:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442992896:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203641.729447:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203641.729456:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203641/real 1788203641] req@ffff90234519d880 x1875064442992896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203641.729466:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064442992896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203641.729474:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203641.729477:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203641.729480:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442992896:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203641.729486:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442993024:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203641.729503:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203641.729513:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203641/real 1788203641] req@ffff90234519ed80 x1875064442993024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203641.729525:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064442993024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203641.729532:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203641.729534:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203641.729537:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442993024:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203642.561024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203643.585065:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203644.608991:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203645.633026:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203646.657035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203646.848993:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203646.849002:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203646.849006:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203646.849015:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2918 00000100:00080000:0.0:1788203646.849025:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203646.849032:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203646.849066:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064442993152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203646.849109:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203646.849113:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203646.849115:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203646.849119:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2918 00000100:00080000:0.0:1788203646.849122:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203646.849127:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203646.849136:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203646.849147:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442993152:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203646.849151:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064442993280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203646.849168:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203646.849171:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203646.849173:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203646.849176:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2918 00000100:00080000:0.0:1788203646.849180:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203646.849184:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203646.849188:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203646.849198:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064442993408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203646.849201:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788203646.849211:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203646.849216:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203646.849217:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203646.849221:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2918 00000100:00080000:0.0:1788203646.849224:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203646.849228:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203646.849232:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203646.849234:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203646/real 1788203646] req@ffff9023450bbb80 x1875064442993152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203646.849243:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064442993536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203646.849250:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064442993152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203646.849265:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203646.849269:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203646.849274:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442993152:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203646.849296:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442993280:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203646.849313:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203646.849321:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203646/real 1788203646] req@ffff9023450b8e00 x1875064442993280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203646.849328:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064442993280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203646.849333:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203646.849335:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203646.849337:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442993280:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203646.849341:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442993408:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203646.849364:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203646.849370:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203646/real 1788203646] req@ffff9023450ba680 x1875064442993408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203646.849377:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064442993408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203646.849382:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203646.849384:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203646.849385:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442993408:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203646.849388:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442993536:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203646.849397:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203646.849403:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203646/real 1788203646] req@ffff9023450bb480 x1875064442993536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203646.849413:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064442993536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203646.849420:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203646.849422:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203646.849425:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442993536:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203647.681037:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203648.705030:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203649.729004:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203650.753051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203651.777019:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203651.968931:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203651.968939:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203651.968942:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203651.968949:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2923 00000100:00080000:0.0:1788203651.968958:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203651.968964:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203651.969007:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064442993664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203651.969034:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203651.969037:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203651.969039:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203651.969043:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2923 00000100:00080000:0.0:1788203651.969047:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203651.969050:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203651.969057:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203651.969067:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064442993792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203651.969074:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203651.969077:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203651.969078:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788203651.969082:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442993664:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203651.969090:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2923 00000100:00080000:0.0:1788203651.969093:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203651.969096:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203651.969100:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203651.969108:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064442993920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203651.969114:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203651.969117:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203651.969118:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203651.969121:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2923 00000100:00080000:0.0:1788203651.969123:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203651.969126:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203651.969128:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788203651.969134:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203651.969137:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064442994048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203651.969173:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203651/real 1788203651] req@ffff9023450b9500 x1875064442993664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203651.969187:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064442993664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203651.969195:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203651.969197:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203651.969200:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442993664:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203651.969217:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442993792:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203651.969232:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203651.969239:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203651/real 1788203651] req@ffff9023450b8380 x1875064442993792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203651.969247:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064442993792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203651.969251:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203651.969254:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203651.969256:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442993792:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203651.969259:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442993920:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203651.969268:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203651.969273:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203651/real 1788203651] req@ffff9023450b8a80 x1875064442993920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203651.969280:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064442993920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203651.969285:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203651.969286:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203651.969288:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442993920:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203651.969291:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442994048:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203651.969302:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203651.969310:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203651/real 1788203651] req@ffff9023450bb800 x1875064442994048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203651.969321:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064442994048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203651.969328:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203651.969332:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203651.969335:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442994048:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203652.801050:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203653.825043:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203654.849042:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203655.873019:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203656.897067:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203657.089017:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203657.089028:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203657.089032:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203657.089044:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2928 00000100:00080000:1.0:1788203657.089055:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203657.089062:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203657.089099:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064442994176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203657.089135:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203657.089139:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203657.089142:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203657.089146:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2928 00000100:00080000:1.0:1788203657.089150:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203657.089155:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203657.089165:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203657.089179:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064442994304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203657.089192:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203657.089196:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203657.089200:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788203657.089201:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442994176:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203657.089219:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2928 00000100:00080000:1.0:1788203657.089225:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203657.089230:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203657.089236:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203657.089250:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064442994432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203657.089260:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788203657.089262:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203657.089267:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203657.089270:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203657.089275:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2928 00000100:00080000:1.0:1788203657.089279:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203657.089284:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203657.089289:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203657.089297:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203657/real 1788203657] req@ffff90234519c000 x1875064442994176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203657.089303:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442994560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203657.089318:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064442994176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203657.089327:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203657.089330:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203657.089334:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442994176:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203657.089352:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442994304:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203657.089366:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203657.089375:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203657/real 1788203657] req@ffff90234519df80 x1875064442994304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203657.089383:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064442994304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203657.089388:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203657.089390:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203657.089392:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442994304:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203657.089396:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442994432:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203657.089407:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203657.089413:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203657/real 1788203657] req@ffff90234519fb80 x1875064442994432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203657.089420:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064442994432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203657.089425:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203657.089427:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203657.089429:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442994432:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203657.089432:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442994560:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203657.089441:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203657.089446:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203657/real 1788203657] req@ffff90234519c700 x1875064442994560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203657.089454:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442994560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203657.089463:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203657.089465:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203657.089467:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442994560:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203657.921047:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203658.945019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203659.969027:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203660.993054:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203662.017013:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203662.208976:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203662.208984:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203662.208988:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203662.208997:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2933 00000100:00080000:1.0:1788203662.209013:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203662.209024:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203662.209063:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442994688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203662.209091:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203662.209094:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203662.209097:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203662.209101:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2933 00000100:00080000:1.0:1788203662.209106:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203662.209111:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203662.209120:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203662.209134:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442994688:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203662.209135:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442994816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203662.209152:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203662.209155:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203662.209157:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203662.209162:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2933 00000100:00080000:1.0:1788203662.209166:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203662.209170:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203662.209175:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203662.209190:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442994944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203662.209201:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203662.209205:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203662.209208:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788203662.209212:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788203662.209217:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2933 00000100:00080000:1.0:1788203662.209223:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203662.209228:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203662.209233:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203662.209247:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442995072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203662.209263:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203662/real 1788203662] req@ffff90234519ea00 x1875064442994688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203662.209281:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442994688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203662.209291:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203662.209296:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203662.209302:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442994688:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203662.209331:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442994816:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203662.209374:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203662.209385:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203662/real 1788203662] req@ffff90234519f100 x1875064442994816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203662.209395:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442994816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203662.209402:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203662.209404:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203662.209407:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442994816:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203662.209412:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442994944:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203662.209431:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203662.209445:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203662/real 1788203662] req@ffff90234519c380 x1875064442994944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203662.209460:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442994944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203662.209470:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203662.209474:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203662.209480:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442994944:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203662.209488:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442995072:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203662.209517:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203662.209532:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203662/real 1788203662] req@ffff90234519f480 x1875064442995072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203662.209547:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442995072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203662.209556:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203662.209559:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203662.209563:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442995072:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203663.041031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203664.065027:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203665.089027:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203666.113031:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203667.137043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203667.328982:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203667.329000:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203667.329005:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203667.329015:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2938 00000100:00080000:1.0:1788203667.329026:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203667.329034:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203667.329070:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064442995200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203667.329098:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203667.329102:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203667.329104:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203667.329109:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2938 00000100:00080000:1.0:1788203667.329113:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203667.329117:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203667.329128:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203667.329143:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064442995328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203667.329155:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788203667.329156:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442995200:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203667.329161:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203667.329165:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203667.329171:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2938 00000100:00080000:1.0:1788203667.329174:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203667.329178:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203667.329183:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203667.329195:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064442995456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203667.329205:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203667.329209:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203667.329211:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203667.329216:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2938 00000100:00080000:1.0:1788203667.329220:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203667.329226:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:2.0:1788203667.329227:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:1.0:1788203667.329235:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203667.329246:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064442995584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203667.329274:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203667/real 1788203667] req@ffff90234519dc00 x1875064442995200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203667.329295:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064442995200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203667.329307:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203667.329312:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203667.329316:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442995200:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203667.329341:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442995328:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203667.329367:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203667.329381:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203667/real 1788203667] req@ffff90234519f480 x1875064442995328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203667.329396:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064442995328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203667.329405:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203667.329409:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203667.329413:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442995328:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203667.329419:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442995456:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203667.329441:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203667.329454:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203667/real 1788203667] req@ffff90234519c380 x1875064442995456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203667.329467:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064442995456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203667.329483:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203667.329486:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203667.329489:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442995456:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203667.329495:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442995584:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203667.329516:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203667.329527:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203667/real 1788203667] req@ffff90234519f100 x1875064442995584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203667.329539:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064442995584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203667.329548:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203667.329551:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203667.329555:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442995584:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203668.161013:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203669.185000:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203670.209043:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203671.232997:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203672.257010:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203672.448989:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203672.448997:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203672.449000:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203672.449008:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2943 00000100:00080000:2.0:1788203672.449016:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203672.449022:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203672.449052:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442995712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203672.449077:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203672.449080:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203672.449081:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203672.449083:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2943 00000100:00080000:2.0:1788203672.449086:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203672.449088:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203672.449094:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203672.449102:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442995840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203672.449109:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203672.449111:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203672.449112:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203672.449114:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2943 00000100:00080000:2.0:1788203672.449115:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203672.449117:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203672.449120:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203672.449125:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442995968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203672.449132:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442995712:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203672.449133:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203672.449139:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203672.449141:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203672.449144:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2943 00000100:00080000:2.0:1788203672.449145:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203672.449148:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203672.449151:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203672.449158:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442996096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203672.449192:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203672.449221:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203672/real 1788203672] req@ffff90234598c380 x1875064442995712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203672.449233:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442995712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203672.449240:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203672.449243:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203672.449245:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442995712:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203672.449249:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442995840:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203672.449261:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203672.449267:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203672/real 1788203672] req@ffff90234598d180 x1875064442995840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203672.449274:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442995840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203672.449279:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203672.449280:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203672.449282:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442995840:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203672.449297:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442995968:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203672.449307:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203672.449312:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203672/real 1788203672] req@ffff90234598e300 x1875064442995968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203672.449319:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442995968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203672.449323:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203672.449325:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203672.449326:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442995968:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203672.449329:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442996096:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203672.449337:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203672.449342:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203672/real 1788203672] req@ffff90234598df80 x1875064442996096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203672.449348:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442996096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203672.449356:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203672.449357:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203672.449359:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442996096:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203673.280986:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203674.305048:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203675.329074:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203676.353090:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203677.377164:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203677.569122:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203677.569133:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203677.569137:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203677.569148:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2949 00000100:00080000:0.0:1788203677.569160:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203677.569168:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203677.569208:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442996224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203677.569239:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203677.569243:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203677.569245:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203677.569250:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2949 00000100:00080000:0.0:1788203677.569254:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203677.569259:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203677.569270:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203677.569288:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442996224:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203677.569289:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442996352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203677.569308:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203677.569311:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203677.569313:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203677.569318:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2949 00000100:00080000:0.0:1788203677.569322:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203677.569327:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203677.569332:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203677.569346:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442996480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203677.569357:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203677.569362:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1788203677.569366:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788203677.569367:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203677.569374:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2949 00000100:00080000:0.0:1788203677.569380:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203677.569385:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203677.569391:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203677.569405:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442996608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203677.569415:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203677/real 1788203677] req@ffff9023450bad80 x1875064442996224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203677.569456:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442996224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203677.569465:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203677.569469:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203677.569473:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442996224:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203677.569493:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442996352:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203677.569511:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203677.569520:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203677/real 1788203677] req@ffff9023450b9c00 x1875064442996352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203677.569529:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442996352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203677.569562:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203677.569565:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203677.569567:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442996352:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203677.569572:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442996480:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203677.569592:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203677.569605:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203677/real 1788203677] req@ffff9023450b9f80 x1875064442996480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203677.569619:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442996480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203677.569628:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203677.569631:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203677.569635:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442996480:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203677.569641:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442996608:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203677.569663:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203677.569675:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203677/real 1788203677] req@ffff9023450b8700 x1875064442996608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203677.569688:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442996608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203677.569696:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203677.569699:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203677.569703:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442996608:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203678.401037:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203679.425012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203680.449033:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203681.473016:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203682.497010:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203682.688966:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203682.688974:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203682.688978:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203682.688987:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2954 00000100:00080000:2.0:1788203682.688996:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203682.689002:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203682.689045:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064442996736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203682.689068:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203682.689072:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203682.689073:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203682.689078:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2954 00000100:00080000:2.0:1788203682.689081:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203682.689085:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203682.689093:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203682.689104:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064442996864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203682.689112:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442996736:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203682.689123:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203682.689127:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203682.689129:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203682.689133:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2954 00000100:00080000:2.0:1788203682.689137:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203682.689140:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203682.689145:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203682.689158:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064442996992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203682.689167:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203682.689170:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203682.689172:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203682.689176:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2954 00000800:00000100:3.0:1788203682.689181:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788203682.689183:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203682.689189:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203682.689194:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203682.689211:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064442997120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203682.689213:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203682/real 1788203682] req@ffff90234598fb80 x1875064442996736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203682.689244:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064442996736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203682.689252:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203682.689255:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203682.689258:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442996736:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203682.689275:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442996864:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203682.689288:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203682.689298:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203682/real 1788203682] req@ffff90234598ed80 x1875064442996864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203682.689309:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064442996864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203682.689317:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203682.689321:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203682.689323:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442996864:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203682.689329:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442996992:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203682.689347:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203682.689354:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203682/real 1788203682] req@ffff90234598f100 x1875064442996992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203682.689361:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064442996992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203682.689370:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203682.689372:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203682.689374:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442996992:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203682.689377:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442997120:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203682.689387:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203682.689392:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203682/real 1788203682] req@ffff90234598df80 x1875064442997120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203682.689399:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064442997120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203682.689403:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203682.689404:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203682.689406:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442997120:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203683.520987:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203684.545085:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203685.569033:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203686.593059:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203687.617059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203687.808978:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203687.808988:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203687.808991:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203687.809000:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2959 00000100:00080000:2.0:1788203687.809018:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203687.809025:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203687.809063:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064442997248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203687.809092:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203687.809098:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203687.809100:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203687.809104:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2959 00000100:00080000:2.0:1788203687.809108:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203687.809112:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203687.809121:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203687.809133:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064442997376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203687.809135:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442997248:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203687.809142:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203687.809145:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203687.809147:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203687.809150:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2959 00000100:00080000:2.0:1788203687.809153:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203687.809156:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203687.809160:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203687.809167:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064442997504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203687.809175:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203687.809178:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203687.809179:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203687.809182:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2959 00000800:00000100:3.0:1788203687.809187:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788203687.809191:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203687.809195:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203687.809199:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203687.809208:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064442997632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203687.809220:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203687/real 1788203687] req@ffff90234598e300 x1875064442997248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203687.809234:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064442997248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203687.809242:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203687.809246:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203687.809249:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442997248:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203687.809270:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442997376:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203687.809286:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203687.809294:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203687/real 1788203687] req@ffff90234598d180 x1875064442997376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203687.809302:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064442997376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203687.809306:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203687.809308:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203687.809310:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442997376:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203687.809313:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442997504:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203687.809323:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203687.809328:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203687/real 1788203687] req@ffff90234598c380 x1875064442997504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203687.809335:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064442997504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203687.809339:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203687.809341:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203687.809342:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442997504:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203687.809345:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442997632:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203687.809354:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203687.809385:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203687/real 1788203687] req@ffff90234598f480 x1875064442997632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203687.809392:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064442997632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203687.809397:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203687.809399:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203687.809400:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442997632:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203688.641001:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203689.665036:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203690.689003:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203691.713039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203692.736984:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203692.928955:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203692.928962:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203692.928965:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203692.928972:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2964 00000100:00080000:2.0:1788203692.928981:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203692.928986:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203692.929023:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064442997760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203692.929043:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203692.929046:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203692.929047:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203692.929050:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2964 00000100:00080000:2.0:1788203692.929052:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203692.929054:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203692.929062:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203692.929074:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064442997888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203692.929085:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203692.929090:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788203692.929091:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442997760:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203692.929092:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203692.929101:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2964 00000100:00080000:2.0:1788203692.929105:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203692.929109:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203692.929113:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203692.929123:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064442998016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203692.929129:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203692.929132:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203692.929133:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203692.929136:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2964 00000100:00080000:2.0:1788203692.929145:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203692.929147:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203692.929150:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203692.929154:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203692.929160:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064442998144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203692.929195:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203692/real 1788203692] req@ffff90234598ce00 x1875064442997760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203692.929208:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064442997760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203692.929216:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203692.929218:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203692.929221:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442997760:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203692.929239:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442997888:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203692.929259:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203692.929266:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203692/real 1788203692] req@ffff90234598c700 x1875064442997888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203692.929290:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064442997888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203692.929304:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203692.929306:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203692.929308:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442997888:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203692.929312:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442998016:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203692.929329:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203692.929338:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203692/real 1788203692] req@ffff90234598dc00 x1875064442998016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203692.929349:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064442998016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203692.929356:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203692.929359:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203692.929362:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442998016:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203692.929368:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442998144:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203692.929385:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203692.929395:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203692/real 1788203692] req@ffff90234598f800 x1875064442998144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203692.929406:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064442998144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203692.929413:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203692.929416:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203692.929419:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442998144:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203693.761073:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203694.785050:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203695.809039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203696.833012:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203697.857047:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203698.048979:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203698.048986:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203698.048989:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203698.048997:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2969 00000100:00080000:0.0:1788203698.049006:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203698.049013:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203698.049046:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442998272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203698.049071:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203698.049074:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203698.049075:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203698.049078:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2969 00000100:00080000:0.0:1788203698.049082:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203698.049086:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203698.049094:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203698.049101:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442998272:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203698.049113:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442998400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203698.049122:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203698.049125:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203698.049128:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203698.049132:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2969 00000100:00080000:0.0:1788203698.049136:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203698.049141:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203698.049147:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788203698.049152:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203698.049169:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442998528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203698.049190:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203698.049193:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203698.049197:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788203698.049197:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203698/real 1788203698] req@ffff9023450baa00 x1875064442998272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203698.049210:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2969 00000100:00080000:0.0:1788203698.049214:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203698.049216:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442998272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203698.049222:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00080000:2.0:1788203698.049227:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788203698.049229:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1788203698.049232:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203698.049236:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442998272:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203698.049245:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442998656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203698.049303:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442998400:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203698.049322:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203698.049334:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203698/real 1788203698] req@ffff9023450b9180 x1875064442998400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203698.049345:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442998400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203698.049353:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203698.049357:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203698.049360:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442998400:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203698.049366:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442998528:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203698.049383:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203698.049390:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203698/real 1788203698] req@ffff9023450b8000 x1875064442998528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203698.049398:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442998528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203698.049404:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203698.049406:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203698.049408:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442998528:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203698.049411:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442998656:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203698.049422:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203698.049427:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203698/real 1788203698] req@ffff9023450ba300 x1875064442998656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203698.049435:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442998656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203698.049440:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203698.049442:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203698.049444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442998656:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203698.881048:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203699.905052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203700.928988:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203701.953037:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203702.977020:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203703.169018:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203703.169032:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203703.169037:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203703.169049:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2974 00000100:00080000:0.0:1788203703.169062:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203703.169070:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203703.169149:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064442998784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203703.169183:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203703.169187:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203703.169189:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203703.169194:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2974 00000100:00080000:0.0:1788203703.169199:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203703.169204:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203703.169216:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203703.169216:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442998784:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203703.169233:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064442998912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203703.169244:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203703.169249:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203703.169250:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203703.169254:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2974 00000100:00080000:0.0:1788203703.169258:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203703.169262:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203703.169267:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203703.169281:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064442999040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203703.169290:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203703.169293:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203703.169295:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203703.169298:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2974 00000100:00080000:0.0:1788203703.169302:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203703.169306:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203703.169311:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203703.169324:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064442999168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203703.169328:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442998912:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203703.169360:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442999040:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203703.169376:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442999168:192.168.201.130@tcp:8:kworker.0 00010000:00010000:3.0:1788203704.001007:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203705.024986:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203706.048975:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788203706.241133:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:1.0:1788203706.241174:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788203706.241204:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788203706.241241:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788203706.241250:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788203706.241258:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:2.0:1788203706.241291:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203703/real 1788203706] req@ffff9023450baa00 x1875064442999168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203706.241316:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064442999168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203706.241326:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203706.241331:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203706.241335:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442999168:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203706.241342:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203703/real 1788203706] req@ffff9023450b8000 x1875064442998912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203706.241360:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064442998912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203706.241367:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203706.241370:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203706.241373:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442998912:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203706.241379:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203703/real 1788203706] req@ffff9023450b9180 x1875064442999040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203706.241388:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064442999040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203706.241394:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203706.241397:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203706.241400:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442999040:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203706.241406:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203703/real 1788203706] req@ffff9023450ba300 x1875064442998784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203706.241416:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064442998784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203706.241423:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203706.241426:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203706.241429:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442998784:192.168.201.130@tcp:250:kworker.0 00010000:00010000:2.0:1788203707.073046:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203708.096983:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203708.289037:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203708.289048:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203708.289052:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203708.289064:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2979 00000100:00080000:0.0:1788203708.289076:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203708.289084:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203708.289118:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064442999296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203708.289142:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203708.289144:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203708.289145:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203708.289148:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2979 00000100:00080000:0.0:1788203708.289151:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203708.289153:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203708.289160:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203708.289167:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064442999424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203708.289174:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203708.289179:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203708.289180:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203708.289182:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2979 00000100:00080000:0.0:1788203708.289184:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203708.289186:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203708.289189:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203708.289195:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064442999552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203708.289201:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442999296:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203708.289203:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203708.289209:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203708.289211:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203708.289214:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2979 00000100:00080000:0.0:1788203708.289216:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203708.289218:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203708.289221:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203708.289228:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064442999680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203708.289286:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203708.289335:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203708/real 1788203708] req@ffff9023450b8700 x1875064442999296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203708.289353:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064442999296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203708.289363:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203708.289367:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203708.289371:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442999296:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203708.289378:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442999424:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203708.289399:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203708.289411:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203708/real 1788203708] req@ffff9023450b9f80 x1875064442999424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203708.289423:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064442999424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203708.289430:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203708.289433:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203708.289436:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442999424:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203708.289459:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442999552:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203708.289490:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203708.289503:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203708/real 1788203708] req@ffff9023450b9c00 x1875064442999552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203708.289523:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064442999552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203708.289532:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203708.289535:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203708.289538:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442999552:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203708.289544:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442999680:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203708.289564:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203708.289575:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203708/real 1788203708] req@ffff9023450bad80 x1875064442999680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203708.289587:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064442999680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203708.289595:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203708.289598:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203708.289601:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442999680:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203709.121081:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203710.144982:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203711.169054:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203712.193018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203713.217050:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203713.409028:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203713.409038:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203713.409042:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203713.409052:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2984 00000100:00080000:1.0:1788203713.409064:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203713.409082:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203713.409114:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064442999808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203713.409133:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203713.409135:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203713.409136:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203713.409139:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2984 00000100:00080000:1.0:1788203713.409141:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203713.409143:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203713.409149:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203713.409157:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064442999936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203713.409164:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203713.409166:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203713.409167:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203713.409171:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2984 00000100:00100000:2.0:1788203713.409171:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442999808:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203713.409174:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203713.409179:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203713.409183:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203713.409189:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443000064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203713.409194:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203713.409196:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203713.409198:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203713.409200:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2984 00000100:00080000:1.0:1788203713.409202:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203713.409204:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203713.409207:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203713.409212:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443000192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203713.409225:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203713.409256:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203713/real 1788203713] req@ffff90234519ea00 x1875064442999808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203713.409272:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064442999808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203713.409279:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203713.409281:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203713.409283:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064442999808:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203713.409298:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442999936:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203713.409311:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203713.409318:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203713/real 1788203713] req@ffff90234519c700 x1875064442999936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203713.409324:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064442999936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203713.409329:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203713.409330:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203713.409332:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064442999936:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203713.409335:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443000064:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203713.409349:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203713.409358:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203713/real 1788203713] req@ffff90234519fb80 x1875064443000064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203713.409369:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443000064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203713.409382:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203713.409385:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203713.409388:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443000064:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203713.409394:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443000192:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203713.409411:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203713.409418:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203713/real 1788203713] req@ffff90234519df80 x1875064443000192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203713.409425:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443000192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203713.409430:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203713.409432:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203713.409433:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443000192:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203714.240993:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203715.265013:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203716.288983:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203717.312988:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203718.337005:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203718.529003:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203718.529013:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203718.529017:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203718.529026:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2990 00000100:00080000:3.0:1788203718.529036:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203718.529044:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203718.529073:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443000320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203718.529093:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203718.529095:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203718.529096:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203718.529099:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2990 00000100:00080000:3.0:1788203718.529101:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203718.529104:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203718.529111:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203718.529119:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443000448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203718.529125:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788203718.529130:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443000320:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203718.529130:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203718.529135:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203718.529139:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2990 00000100:00080000:3.0:1788203718.529141:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203718.529143:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203718.529145:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203718.529159:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443000576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203718.529165:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203718.529167:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203718.529168:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203718.529170:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2990 00000100:00080000:3.0:1788203718.529174:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203718.529176:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203718.529179:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788203718.529182:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203718.529190:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443000704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203718.529217:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203718/real 1788203718] req@ffff902352689180 x1875064443000320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203718.529230:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443000320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203718.529242:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203718.529245:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203718.529248:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443000320:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203718.529266:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443000448:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203718.529279:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203718.529286:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203718/real 1788203718] req@ffff902352688700 x1875064443000448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203718.529294:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443000448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203718.529298:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203718.529300:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203718.529301:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443000448:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203718.529305:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443000576:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203718.529315:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203718.529328:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203718/real 1788203718] req@ffff90235268bb80 x1875064443000576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203718.529336:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443000576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203718.529341:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203718.529344:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203718.529346:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443000576:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203718.529349:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443000704:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203718.529359:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203718.529365:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203718/real 1788203718] req@ffff902352689880 x1875064443000704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203718.529371:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443000704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203718.529375:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203718.529377:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203718.529379:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443000704:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203719.361029:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203720.384985:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203721.409031:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203722.433001:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203723.457054:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203723.648967:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203723.648976:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203723.648980:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203723.648988:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 2995 00000100:00080000:2.0:1788203723.648998:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203723.649004:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203723.649035:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064443000832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203723.649058:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203723.649060:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203723.649061:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203723.649064:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2995 00000100:00080000:2.0:1788203723.649067:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203723.649069:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203723.649075:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203723.649083:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064443000960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203723.649090:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203723.649092:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203723.649093:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203723.649097:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2995 00000100:00100000:3.0:1788203723.649100:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443000832:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203723.649101:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203723.649108:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203723.649113:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203723.649125:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064443001088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203723.649135:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203723.649138:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203723.649140:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203723.649144:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 2995 00000100:00080000:2.0:1788203723.649147:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203723.649150:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203723.649153:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203723.649156:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203723.649165:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443001216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203723.649188:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203723/real 1788203723] req@ffff90234598d880 x1875064443000832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203723.649200:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064443000832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203723.649211:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203723.649214:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203723.649217:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443000832:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203723.649233:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443000960:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203723.649247:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203723.649254:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203723/real 1788203723] req@ffff90234598e680 x1875064443000960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203723.649261:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064443000960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203723.649266:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203723.649268:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203723.649270:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443000960:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203723.649273:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443001088:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203723.649282:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203723.649287:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203723/real 1788203723] req@ffff90234598ca80 x1875064443001088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203723.649293:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064443001088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203723.649298:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203723.649300:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203723.649309:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443001088:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203723.649313:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443001216:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203723.649322:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203723.649328:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203723/real 1788203723] req@ffff90234598f800 x1875064443001216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203723.649334:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443001216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203723.649339:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203723.649341:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203723.649342:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443001216:192.168.201.130@tcp:8:kworker.0 00010000:00010000:3.0:1788203724.480946:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203725.505047:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203726.529029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203727.553057:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203728.577044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203728.769060:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203728.769074:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203728.769080:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203728.769094:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3000 00000100:00080000:3.0:1788203728.769108:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203728.769115:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203728.769157:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443001344/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203728.769193:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203728.769198:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203728.769200:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203728.769205:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3000 00000100:00080000:3.0:1788203728.769210:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203728.769216:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203728.769228:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203728.769239:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443001344:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203728.769248:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443001472/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203728.769257:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203728.769260:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203728.769261:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203728.769264:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3000 00000100:00080000:3.0:1788203728.769268:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203728.769270:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203728.769275:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203728.769285:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443001600/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203728.769292:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203728.769294:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203728.769295:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203728.769298:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3000 00000100:00080000:3.0:1788203728.769301:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203728.769303:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203728.769305:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788203728.769312:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203728.769316:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064443001728/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203728.769362:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203728/real 1788203728] req@ffff902352688380 x1875064443001344/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203728.769379:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443001344/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203728.769388:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203728.769391:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203728.769394:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443001344:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203728.769416:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443001472:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203728.769432:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203728.769441:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203728/real 1788203728] req@ffff90235268ad80 x1875064443001472/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203728.769450:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443001472/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203728.769456:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203728.769458:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203728.769460:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443001472:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203728.769464:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443001600:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203728.769477:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203728.769483:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203728/real 1788203728] req@ffff90235268b480 x1875064443001600/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203728.769491:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443001600/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203728.769496:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203728.769498:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203728.769500:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443001600:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203728.769503:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443001728:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203728.769514:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203728.769519:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203728/real 1788203728] req@ffff902352688a80 x1875064443001728/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203728.769527:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064443001728/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203728.769532:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203728.769534:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203728.769536:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443001728:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203729.601029:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203730.625060:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203731.649001:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203732.673022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203733.697130:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203733.888960:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203733.888971:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203733.888975:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203733.888987:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3005 00000100:00080000:0.0:1788203733.888999:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203733.889015:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203733.889048:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443001856/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203733.889076:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203733.889080:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203733.889082:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203733.889087:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3005 00000100:00080000:0.0:1788203733.889092:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203733.889098:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203733.889108:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203733.889122:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443001856:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203733.889125:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443001984/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203733.889139:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203733.889142:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203733.889144:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203733.889147:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3005 00000100:00080000:0.0:1788203733.889151:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203733.889154:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203733.889157:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203733.889165:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443002112/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203733.889173:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203733.889176:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203733.889178:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203733.889181:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3005 00000100:00080000:0.0:1788203733.889185:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203733.889188:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203733.889191:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203733.889213:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443002240/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203733.889270:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203733.889305:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203733/real 1788203733] req@ffff9023450bb800 x1875064443001856/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203733.889324:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443001856/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203733.889452:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203733.889458:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203733.889464:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443001856:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203733.889524:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443001984:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203733.889546:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203733.889583:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203733/real 1788203733] req@ffff9023450b8a80 x1875064443001984/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203733.889595:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443001984/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203733.889601:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203733.889604:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203733.889606:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443001984:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203733.889611:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443002112:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203733.889626:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203733.889634:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203733/real 1788203733] req@ffff9023450b8380 x1875064443002112/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203733.889646:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443002112/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203733.889652:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203733.889653:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203733.889655:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443002112:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203733.889659:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443002240:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203733.889670:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203733.889675:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203733/real 1788203733] req@ffff9023450b9500 x1875064443002240/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203733.889683:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443002240/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203733.889688:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203733.889690:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203733.889692:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443002240:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203734.721057:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203735.745000:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203736.769025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203737.793018:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203738.817035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203739.008981:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203739.008990:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203739.008993:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203739.009001:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3010 00000100:00080000:3.0:1788203739.009010:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203739.009015:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203739.009053:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443002368/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203739.009075:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203739.009078:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203739.009080:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203739.009083:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3010 00000100:00080000:3.0:1788203739.009085:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203739.009087:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203739.009095:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203739.009106:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064443002496/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203739.009113:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443002368:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203739.009117:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203739.009125:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203739.009126:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203739.009130:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3010 00000100:00080000:3.0:1788203739.009133:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203739.009136:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203739.009140:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203739.009149:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443002624/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203739.009157:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203739.009160:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203739.009161:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203739.009163:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3010 00000100:00080000:3.0:1788203739.009167:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788203739.009169:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788203739.009172:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203739.009177:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203739.009185:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443002752/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203739.009207:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203739/real 1788203739] req@ffff90235268aa00 x1875064443002368/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203739.009221:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443002368/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203739.009228:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203739.009231:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203739.009234:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443002368:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203739.009253:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443002496:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203739.009275:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203739.009289:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203739/real 1788203739] req@ffff902352688a80 x1875064443002496/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203739.009302:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064443002496/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203739.009309:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203739.009311:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203739.009313:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443002496:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203739.009318:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443002624:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203739.009333:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203739.009340:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203739/real 1788203739] req@ffff90235268b480 x1875064443002624/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203739.009352:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443002624/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203739.009357:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203739.009359:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203739.009361:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443002624:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203739.009365:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443002752:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203739.009374:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203739.009380:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203739/real 1788203739] req@ffff90235268ad80 x1875064443002752/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203739.009387:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443002752/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203739.009392:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203739.009394:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203739.009395:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443002752:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203739.841021:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203740.865005:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203741.889015:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203742.913022:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203743.937096:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203744.129038:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203744.129046:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203744.129050:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203744.129059:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3015 00000100:00080000:1.0:1788203744.129076:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203744.129082:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203744.129116:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443002880/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203744.129141:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203744.129144:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203744.129145:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203744.129148:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3015 00000100:00080000:1.0:1788203744.129150:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203744.129153:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203744.129168:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203744.129183:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443003008/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203744.129199:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203744.129206:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788203744.129207:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443002880:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203744.129209:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203744.129225:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3015 00000100:00080000:1.0:1788203744.129229:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203744.129232:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203744.129238:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203744.129259:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443003136/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203744.129267:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203744.129270:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203744.129272:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203744.129275:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3015 00000100:00080000:1.0:1788203744.129278:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203744.129280:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203744.129283:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203744.129289:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443003264/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203744.129313:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203744.129365:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203744/real 1788203744] req@ffff90234519c000 x1875064443002880/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203744.129410:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443002880/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203744.129422:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203744.129425:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203744.129428:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443002880:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203744.129455:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443003008:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203744.129476:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203744.129487:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203744/real 1788203744] req@ffff90234519ed80 x1875064443003008/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203744.129495:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443003008/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203744.129501:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203744.129504:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203744.129506:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443003008:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203744.129511:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443003136:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203744.129523:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203744.129529:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203744/real 1788203744] req@ffff90234519d880 x1875064443003136/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203744.129537:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443003136/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203744.129543:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203744.129545:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203744.129547:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443003136:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203744.129552:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443003264:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203744.129562:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203744.129577:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203744/real 1788203744] req@ffff90234519e300 x1875064443003264/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203744.129589:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443003264/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203744.129594:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203744.129597:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203744.129599:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443003264:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203744.961045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203745.985105:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203747.009084:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203748.033004:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203749.057014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203749.248999:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203749.249008:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203749.249013:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203749.249024:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3020 00000100:00080000:0.0:1788203749.249035:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203749.249042:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203749.249078:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443003392/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203749.249106:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203749.249110:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203749.249112:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203749.249117:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3020 00000100:00080000:0.0:1788203749.249120:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203749.249124:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203749.249134:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203749.249146:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064443003520/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203749.249153:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443003392:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203749.249165:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203749.249167:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203749.249169:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203749.249172:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3020 00000100:00080000:0.0:1788203749.249175:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203749.249178:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203749.249183:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203749.249197:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064443003648/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203749.249207:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203749.249211:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1788203749.249214:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788203749.249215:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203749.249223:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3020 00000100:00080000:0.0:1788203749.249228:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203749.249233:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203749.249239:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203749.249253:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203749/real 1788203749] req@ffff9023450bb480 x1875064443003392/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203749.249257:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064443003776/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203749.249275:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443003392/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203749.249284:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203749.249287:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203749.249303:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443003392:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203749.249325:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443003520:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203749.249343:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203749.249353:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203749/real 1788203749] req@ffff9023450ba680 x1875064443003520/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203749.249361:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064443003520/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203749.249367:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203749.249369:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203749.249372:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443003520:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203749.249375:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443003648:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203749.249387:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203749.249393:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203749/real 1788203749] req@ffff9023450b8e00 x1875064443003648/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203749.249400:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064443003648/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203749.249405:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203749.249407:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203749.249409:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443003648:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203749.249412:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443003776:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203749.249421:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203749.249427:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203749/real 1788203749] req@ffff9023450bbb80 x1875064443003776/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203749.249434:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064443003776/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203749.249439:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203749.249441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203749.249443:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443003776:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203750.081111:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203751.104998:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203752.129041:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203753.153060:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203754.177077:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203754.369027:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203754.369041:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203754.369046:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203754.369059:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3025 00000100:00080000:3.0:1788203754.369073:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203754.369082:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203754.369132:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443003904/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203754.369165:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203754.369169:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203754.369171:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203754.369176:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3025 00000100:00080000:3.0:1788203754.369180:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203754.369184:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203754.369195:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203754.369224:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443004032/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203754.369237:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788203754.369246:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443003904:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203754.369247:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203754.369251:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203754.369258:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3025 00000100:00080000:3.0:1788203754.369262:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203754.369267:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203754.369273:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203754.369286:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443004160/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203754.369296:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203754.369300:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203754.369302:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203754.369306:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3025 00000100:00080000:3.0:1788203754.369310:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203754.369314:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203754.369319:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788203754.369321:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203754.369336:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443004288/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203754.369369:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203754/real 1788203754] req@ffff902352688380 x1875064443003904/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203754.369385:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443003904/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203754.369394:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203754.369397:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203754.369401:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443003904:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203754.369423:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443004032:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203754.369439:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203754.369448:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203754/real 1788203754] req@ffff902352689880 x1875064443004032/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203754.369456:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443004032/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203754.369462:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203754.369465:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203754.369468:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443004032:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203754.369472:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443004160:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203754.369483:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203754.369489:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203754/real 1788203754] req@ffff90235268bb80 x1875064443004160/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203754.369497:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443004160/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203754.369503:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203754.369505:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203754.369506:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443004160:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203754.369510:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443004288:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203754.369520:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203754.369526:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203754/real 1788203754] req@ffff902352688700 x1875064443004288/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203754.369533:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443004288/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203754.369544:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203754.369546:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203754.369548:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443004288:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203755.201081:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203756.225101:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203757.249099:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203758.273055:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203759.297032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203759.489030:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203759.489039:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203759.489042:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203759.489051:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3031 00000100:00080000:3.0:1788203759.489062:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203759.489068:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203759.489101:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443004416/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203759.489136:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203759.489139:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203759.489142:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203759.489148:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3031 00000100:00080000:3.0:1788203759.489152:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203759.489156:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203759.489164:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443004416:192.168.201.130@tcp:250:kworker.0 00000008:00100000:3.0:1788203759.489167:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203759.489186:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443004544/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203759.489198:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203759.489202:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203759.489204:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203759.489208:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3031 00000100:00080000:3.0:1788203759.489213:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788203759.489219:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788203759.489220:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203759.489229:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203759.489243:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443004672/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203759.489254:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788203759.489258:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203759/real 1788203759] req@ffff902352689180 x1875064443004416/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203759.489260:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203759.489264:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203759.489275:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3031 00000100:00080000:3.0:1788203759.489281:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203759.489285:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443004416/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203759.489290:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203759.489300:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1788203759.489304:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203759.489309:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203759.489316:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443004800/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203759.489317:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443004416:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203759.489342:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443004544:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203759.489359:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203759.489368:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203759/real 1788203759] req@ffff902352689c00 x1875064443004544/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203759.489378:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443004544/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203759.489388:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203759.489391:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203759.489395:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443004544:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203759.489401:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443004672:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203759.489422:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203759.489434:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203759/real 1788203759] req@ffff90235268a680 x1875064443004672/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203759.489448:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443004672/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203759.489454:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203759.489456:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203759.489458:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443004672:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203759.489462:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443004800:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203759.489475:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203759.489482:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203759/real 1788203759] req@ffff902352688e00 x1875064443004800/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203759.489489:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443004800/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203759.489494:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203759.489496:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203759.489498:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443004800:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203760.321022:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203761.345002:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203762.369024:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203763.393002:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203764.417056:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203764.608965:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203764.608975:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203764.608979:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203764.608990:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3036 00000100:00080000:2.0:1788203764.609001:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203764.609008:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203764.609082:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443004928/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203764.609112:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203764.609116:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203764.609119:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203764.609124:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3036 00000100:00080000:2.0:1788203764.609129:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203764.609134:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203764.609142:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443004928:192.168.201.130@tcp:250:kworker.0 00000008:00100000:2.0:1788203764.609143:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203764.609162:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443005056/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203764.609174:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203764.609178:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203764.609180:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203764.609185:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3036 00000100:00080000:2.0:1788203764.609197:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203764.609201:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:3.0:1788203764.609207:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:2.0:1788203764.609208:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203764.609225:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443005184/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203764.609235:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203764.609239:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203764.609242:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203764.609247:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3036 00000100:00100000:3.0:1788203764.609252:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203764/real 1788203764] req@ffff90234598dc00 x1875064443004928/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203764.609253:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203764.609262:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203764.609272:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203764.609285:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443004928/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203764.609287:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443005312/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203764.609321:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203764.609326:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203764.609329:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443004928:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203764.609362:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443005056:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203764.609381:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203764.609391:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203764/real 1788203764] req@ffff90234598c700 x1875064443005056/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203764.609406:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443005056/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203764.609415:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203764.609419:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203764.609423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443005056:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203764.609430:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443005184:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203764.609451:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203764.609460:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203764/real 1788203764] req@ffff90234598ce00 x1875064443005184/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203764.609473:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443005184/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203764.609482:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203764.609485:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203764.609489:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443005184:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203764.609496:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443005312:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203764.609516:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203764.609525:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203764/real 1788203764] req@ffff90234598f480 x1875064443005312/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203764.609533:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443005312/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203764.609539:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203764.609541:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203764.609543:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443005312:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203765.441076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203766.465112:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203767.489045:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203768.513030:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203769.537101:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203769.729039:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203769.729051:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203769.729056:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203769.729076:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3041 00000100:00080000:2.0:1788203769.729095:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203769.729106:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203769.729154:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443005440/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203769.729187:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203769.729190:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203769.729192:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203769.729196:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3041 00000100:00080000:2.0:1788203769.729199:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203769.729203:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203769.729213:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203769.729226:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443005568/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203769.729240:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203769.729243:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203769.729245:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788203769.729250:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443005440:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203769.729251:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3041 00000100:00080000:2.0:1788203769.729256:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203769.729263:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203769.729268:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203769.729278:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443005696/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203769.729285:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203769.729288:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203769.729290:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203769.729293:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3041 00000100:00080000:2.0:1788203769.729296:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203769.729298:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203769.729301:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203769.729309:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443005824/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203769.729394:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443005568:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203769.729432:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443005696:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203769.729456:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443005824:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203770.561044:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203771.585019:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203772.609123:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788203772.801351:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788203772.801388:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788203772.801403:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788203772.801541:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788203772.801544:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203769/real 1788203772] req@ffff90234598c380 x1875064443005440/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1788203772.801565:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788203772.801586:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788203772.801612:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443005440/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203772.801643:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203772.801648:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203772.801654:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443005440:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203772.801686:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203769/real 1788203772] req@ffff90234598d180 x1875064443005568/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203772.801703:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443005568/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203772.801713:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203772.801717:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203772.801721:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443005568:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203772.801729:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203769/real 1788203772] req@ffff90234598e300 x1875064443005696/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203772.801742:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443005696/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203772.801750:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203772.801754:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203772.801757:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443005696:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203772.801765:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203769/real 1788203772] req@ffff90234598df80 x1875064443005824/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203772.801776:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443005824/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203772.801784:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203772.801788:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203772.801791:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443005824:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203773.632983:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203774.657035:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203774.849015:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203774.849027:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203774.849031:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203774.849042:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3046 00000100:00080000:3.0:1788203774.849060:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203774.849070:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203774.849123:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443005952/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203774.849155:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203774.849160:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203774.849162:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203774.849167:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3046 00000100:00080000:3.0:1788203774.849180:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203774.849186:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203774.849200:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203774.849216:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443005952:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203774.849219:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443006080/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203774.849237:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203774.849241:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203774.849243:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203774.849248:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3046 00000100:00080000:3.0:1788203774.849253:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203774.849258:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203774.849264:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203774.849279:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443006208/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203774.849296:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203774.849300:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203774.849303:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203774.849307:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3046 00000100:00080000:3.0:1788203774.849310:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203774.849315:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:0.0:1788203774.849321:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:3.0:1788203774.849322:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203774.849337:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064443006336/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203774.849389:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203774/real 1788203774] req@ffff902352689500 x1875064443005952/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203774.849416:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443005952/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203774.849430:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203774.849436:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203774.849440:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443005952:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203774.849480:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443006080:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788203774.849508:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203774.849525:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203774/real 1788203774] req@ffff90235268b800 x1875064443006080/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203774.849541:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443006080/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203774.849551:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203774.849555:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203774.849559:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443006080:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788203774.849566:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443006208:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203774.849590:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203774.849603:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203774/real 1788203774] req@ffff90235268a300 x1875064443006208/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203774.849617:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443006208/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203774.849627:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203774.849631:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203774.849634:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443006208:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788203774.849641:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443006336:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788203774.849663:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203774.849677:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203774/real 1788203774] req@ffff90235268b100 x1875064443006336/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203774.849690:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064443006336/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203774.849700:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203774.849711:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788203774.849715:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443006336:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203775.681159:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203776.705058:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203777.729036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203778.753068:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203779.777120:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203779.969009:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203779.969021:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203779.969026:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203779.969037:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3051 00000100:00080000:0.0:1788203779.969050:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203779.969058:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203779.969115:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064443006464/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203779.969158:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203779.969163:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203779.969165:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203779.969170:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3051 00000100:00080000:0.0:1788203779.969175:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203779.969180:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203779.969192:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203779.969211:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064443006592/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203779.969215:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443006464:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203779.969232:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203779.969236:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203779.969238:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203779.969242:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3051 00000100:00080000:0.0:1788203779.969246:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203779.969249:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203779.969255:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203779.969267:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064443006720/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203779.969274:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203779.969277:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203779.969278:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203779.969281:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3051 00000100:00080000:0.0:1788203779.969283:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203779.969286:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203779.969295:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203779.969302:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788203779.969306:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443006848/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203779.969370:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203779/real 1788203779] req@ffff9023450bbb80 x1875064443006464/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203779.969394:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064443006464/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203779.969407:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203779.969412:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203779.969416:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443006464:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203779.969442:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443006592:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203779.969463:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203779.969475:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203779/real 1788203779] req@ffff9023450b8e00 x1875064443006592/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203779.969488:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064443006592/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203779.969497:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203779.969501:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203779.969505:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443006592:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203779.969511:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443006720:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203779.969527:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203779.969536:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203779/real 1788203779] req@ffff9023450ba680 x1875064443006720/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203779.969547:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064443006720/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203779.969556:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203779.969559:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203779.969562:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443006720:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203779.969567:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443006848:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203779.969583:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203779.969594:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203779/real 1788203779] req@ffff9023450bb480 x1875064443006848/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203779.969606:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443006848/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203779.969621:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203779.969624:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203779.969628:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443006848:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203780.801030:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203781.825030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203782.849041:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203783.873016:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203784.897102:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203785.089046:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203785.089061:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203785.089066:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203785.089079:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3056 00000100:00080000:1.0:1788203785.089107:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203785.089117:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203785.089168:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064443006976/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203785.089200:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203785.089205:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203785.089207:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203785.089212:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3056 00000100:00080000:1.0:1788203785.089217:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203785.089222:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203785.089233:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203785.089247:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443006976:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203785.089253:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443007104/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203785.089266:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203785.089271:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203785.089273:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203785.089278:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3056 00000100:00080000:1.0:1788203785.089282:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203785.089288:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203785.089293:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203785.089306:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443007232/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203785.089317:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203785.089322:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203785.089324:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788203785.089327:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788203785.089332:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3056 00000100:00080000:1.0:1788203785.089338:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203785.089343:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203785.089349:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203785.089363:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443007360/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203785.089379:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203785/real 1788203785] req@ffff90234519f800 x1875064443006976/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203785.089399:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064443006976/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203785.089412:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203785.089418:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203785.089423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443006976:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203785.089459:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443007104:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203785.089486:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203785.089501:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203785/real 1788203785] req@ffff90234519e300 x1875064443007104/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203785.089510:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443007104/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203785.089516:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203785.089519:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203785.089521:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443007104:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203785.089526:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443007232:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203785.089542:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203785.089549:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203785/real 1788203785] req@ffff90234519d880 x1875064443007232/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203785.089557:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443007232/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203785.089563:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203785.089565:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203785.089566:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443007232:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203785.089570:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443007360:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203785.089581:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203785.089586:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203785/real 1788203785] req@ffff90234519ed80 x1875064443007360/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203785.089594:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443007360/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203785.089598:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203785.089600:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203785.089602:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443007360:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203785.921061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203786.945101:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203787.969024:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203788.993077:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203790.017005:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203790.209005:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203790.209017:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203790.209021:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203790.209031:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3061 00000100:00080000:2.0:1788203790.209042:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203790.209049:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203790.209085:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064443007488/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203790.209108:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203790.209111:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203790.209113:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203790.209116:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3061 00000100:00080000:2.0:1788203790.209119:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203790.209122:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203790.209130:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203790.209139:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064443007616/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203790.209148:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203790.209150:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203790.209152:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203790.209156:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3061 00000100:00100000:3.0:1788203790.209157:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443007488:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203790.209159:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203790.209165:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203790.209169:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203790.209178:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064443007744/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203790.209184:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203790.209187:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203790.209188:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203790.209191:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3061 00000100:00080000:2.0:1788203790.209206:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203790.209208:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203790.209211:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203790.209216:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203790.209223:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443007872/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203790.209260:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203790/real 1788203790] req@ffff90234598f100 x1875064443007488/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203790.209275:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064443007488/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203790.209283:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203790.209286:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203790.209289:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443007488:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203790.209307:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443007616:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203790.209322:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203790.209330:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203790/real 1788203790] req@ffff90234598ed80 x1875064443007616/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203790.209338:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064443007616/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203790.209344:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203790.209346:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203790.209348:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443007616:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203790.209352:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443007744:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203790.209362:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203790.209368:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203790/real 1788203790] req@ffff90234598fb80 x1875064443007744/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203790.209375:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064443007744/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203790.209385:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203790.209387:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203790.209389:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443007744:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203790.209392:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443007872:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203790.209401:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203790.209407:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203790/real 1788203790] req@ffff90234598df80 x1875064443007872/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203790.209413:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443007872/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203790.209418:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203790.209420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203790.209422:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443007872:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203791.041019:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203792.065100:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203793.089099:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203794.113083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203795.137101:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203795.329036:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203795.329047:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203795.329052:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203795.329062:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3066 00000100:00080000:1.0:1788203795.329075:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203795.329082:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203795.329132:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443008000/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203795.329161:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203795.329164:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203795.329166:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203795.329169:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3066 00000100:00080000:1.0:1788203795.329172:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203795.329176:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203795.329194:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203795.329205:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443008128/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203795.329214:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443008000:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203795.329215:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203795.329223:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203795.329226:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203795.329231:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3066 00000100:00080000:1.0:1788203795.329234:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203795.329239:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203795.329245:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203795.329261:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443008256/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203795.329272:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203795.329276:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203795.329279:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203795.329283:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3066 00000100:00080000:1.0:1788203795.329287:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203795.329290:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:3.0:1788203795.329296:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:1.0:1788203795.329298:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203795.329310:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443008384/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203795.329359:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203795/real 1788203795] req@ffff90234519c000 x1875064443008000/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203795.329378:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443008000/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203795.329386:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203795.329390:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203795.329393:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443008000:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203795.329417:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443008128:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203795.329434:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203795.329443:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203795/real 1788203795] req@ffff90234519df80 x1875064443008128/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203795.329452:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443008128/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203795.329458:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203795.329460:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203795.329463:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443008128:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203795.329473:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443008256:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203795.329487:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203795.329510:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203795/real 1788203795] req@ffff90234519fb80 x1875064443008256/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203795.329519:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443008256/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203795.329525:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203795.329528:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203795.329530:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443008256:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203795.329534:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443008384:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203795.329548:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203795.329556:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203795/real 1788203795] req@ffff90234519c700 x1875064443008384/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203795.329564:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443008384/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203795.329569:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203795.329571:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203795.329573:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443008384:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203796.161113:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203797.185081:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203798.209051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203799.233074:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203800.257096:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203800.449072:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203800.449085:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203800.449090:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203800.449104:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3071 00000100:00080000:2.0:1788203800.449119:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203800.449128:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203800.449182:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443008512/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203800.449219:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203800.449224:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203800.449226:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203800.449232:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3071 00000100:00080000:2.0:1788203800.449236:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203800.449241:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203800.449254:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203800.449270:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443008640/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203800.449281:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443008512:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203800.449285:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203800.449292:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203800.449294:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203800.449300:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3071 00000100:00080000:2.0:1788203800.449304:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203800.449308:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203800.449314:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203800.449330:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443008768/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203800.449340:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203800.449345:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203800.449347:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203800.449350:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3071 00000100:00080000:2.0:1788203800.449369:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203800.449374:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203800.449380:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203800.449384:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203800.449396:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443008896/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203800.449452:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203800/real 1788203800] req@ffff90234598e300 x1875064443008512/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203800.449473:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443008512/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203800.449486:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203800.449491:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203800.449496:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443008512:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203800.449525:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443008640:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203800.449549:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203800.449563:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203800/real 1788203800] req@ffff90234598d180 x1875064443008640/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203800.449587:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443008640/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203800.449608:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203800.449612:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203800.449616:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443008640:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203800.449623:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443008768:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203800.449644:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203800.449655:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203800/real 1788203800] req@ffff90234598c380 x1875064443008768/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203800.449667:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443008768/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203800.449676:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203800.449680:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203800.449683:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443008768:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203800.449690:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443008896:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203800.449708:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203800.449718:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203800/real 1788203800] req@ffff90234598f480 x1875064443008896/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203800.449730:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443008896/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203800.449739:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203800.449742:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203800.449747:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443008896:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203801.281034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203802.305009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203803.329058:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203804.353066:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203805.377030:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203805.568971:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203805.568979:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203805.568982:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203805.568990:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3077 00000100:00080000:2.0:1788203805.568999:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203805.569004:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203805.569037:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443009024/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203805.569065:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203805.569069:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203805.569071:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203805.569075:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3077 00000100:00080000:2.0:1788203805.569078:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203805.569083:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203805.569093:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203805.569102:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443009024:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203805.569108:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443009152/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203805.569117:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203805.569119:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203805.569120:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203805.569123:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3077 00000100:00080000:2.0:1788203805.569126:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203805.569129:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203805.569132:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203805.569140:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443009280/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203805.569146:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203805.569148:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203805.569149:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203805.569154:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3077 00000800:00000100:3.0:1788203805.569154:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788203805.569160:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203805.569162:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203805.569165:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203805.569172:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443009408/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203805.569192:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203805/real 1788203805] req@ffff90234598ce00 x1875064443009024/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203805.569210:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443009024/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203805.569216:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203805.569219:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203805.569221:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443009024:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203805.569240:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443009152:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203805.569256:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203805.569264:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203805/real 1788203805] req@ffff90234598c700 x1875064443009152/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203805.569270:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443009152/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203805.569275:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203805.569277:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203805.569279:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443009152:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203805.569282:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443009280:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203805.569294:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203805.569300:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203805/real 1788203805] req@ffff90234598dc00 x1875064443009280/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203805.569308:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443009280/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203805.569327:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203805.569330:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203805.569333:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443009280:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203805.569341:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443009408:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203805.569355:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203805.569364:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203805/real 1788203805] req@ffff90234598f800 x1875064443009408/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203805.569371:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443009408/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203805.569376:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203805.569379:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203805.569381:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443009408:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203806.401029:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203807.425028:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203808.449128:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203809.473028:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203810.497089:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203810.689046:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203810.689056:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203810.689060:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203810.689069:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3082 00000100:00080000:1.0:1788203810.689080:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203810.689086:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203810.689128:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443009536/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203810.689149:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203810.689152:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203810.689153:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203810.689156:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3082 00000100:00080000:1.0:1788203810.689158:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203810.689161:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203810.689170:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203810.689179:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443009664/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203810.689186:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203810.689188:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203810.689189:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203810.689191:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3082 00000100:00080000:1.0:1788203810.689193:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203810.689198:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203810.689198:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443009536:192.168.201.130@tcp:250:kworker.0 00000008:00100000:1.0:1788203810.689205:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203810.689212:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443009792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203810.689217:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203810.689219:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203810.689220:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203810.689223:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3082 00000100:00080000:1.0:1788203810.689238:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203810.689241:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203810.689244:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203810.689252:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443009920/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203810.689282:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203810.689326:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203810/real 1788203810] req@ffff90234519ea00 x1875064443009536/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203810.689340:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443009536/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203810.689348:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203810.689351:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203810.689354:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443009536:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203810.689374:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443009664:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203810.689390:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203810.689398:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203810/real 1788203810] req@ffff90234519f100 x1875064443009664/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203810.689405:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443009664/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203810.689410:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203810.689413:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203810.689414:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443009664:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203810.689418:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443009792:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203810.689428:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203810.689433:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203810/real 1788203810] req@ffff90234519c380 x1875064443009792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203810.689440:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443009792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203810.689445:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203810.689446:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203810.689448:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443009792:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203810.689452:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443009920:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203810.689461:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203810.689466:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203810/real 1788203810] req@ffff90234519f480 x1875064443009920/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203810.689472:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443009920/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203810.689477:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203810.689478:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203810.689480:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443009920:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203811.521084:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203812.545059:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203813.569043:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203814.593064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203815.617040:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203815.809050:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203815.809061:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203815.809066:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203815.809080:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3087 00000100:00080000:2.0:1788203815.809094:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203815.809102:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203815.809167:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064443010048/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203815.809189:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203815.809192:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203815.809193:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203815.809196:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3087 00000100:00080000:2.0:1788203815.809199:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203815.809202:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203815.809212:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203815.809221:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064443010176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203815.809227:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203815.809230:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203815.809231:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203815.809234:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3087 00000100:00080000:2.0:1788203815.809235:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203815.809237:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203815.809240:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203815.809249:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064443010304/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203815.809253:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443010048:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203815.809256:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203815.809261:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203815.809264:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203815.809266:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3087 00000100:00080000:2.0:1788203815.809268:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203815.809270:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203815.809273:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203815.809280:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443010432/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203815.809334:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203815.809394:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203815/real 1788203815] req@ffff90234598ca80 x1875064443010048/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203815.809414:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064443010048/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203815.809426:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203815.809430:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203815.809434:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443010048:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203815.809441:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443010176:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203815.809463:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203815.809475:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203815/real 1788203815] req@ffff90234598e680 x1875064443010176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203815.809487:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064443010176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203815.809495:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203815.809498:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203815.809502:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443010176:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203815.809530:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443010304:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203815.809551:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203815.809562:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203815/real 1788203815] req@ffff90234598d880 x1875064443010304/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203815.809574:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064443010304/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203815.809582:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203815.809585:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203815.809589:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443010304:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203815.809595:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443010432:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203815.809613:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203815.809623:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203815/real 1788203815] req@ffff90234598f800 x1875064443010432/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203815.809634:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443010432/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203815.809647:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203815.809651:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203815.809654:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443010432:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203816.641077:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203817.665071:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203818.689051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203819.712990:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203820.737027:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203820.928942:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203820.928953:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203820.928957:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203820.928966:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3092 00000100:00080000:2.0:1788203820.928976:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203820.928988:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203820.929021:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443010560/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203820.929068:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203820.929073:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203820.929075:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203820.929080:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3092 00000100:00080000:2.0:1788203820.929083:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203820.929087:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203820.929093:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443010560:192.168.201.130@tcp:250:kworker.0 00000008:00100000:2.0:1788203820.929096:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203820.929112:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443010688/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203820.929124:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203820.929127:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203820.929129:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203820.929139:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3092 00000100:00080000:2.0:1788203820.929144:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788203820.929148:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788203820.929150:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203820.929157:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203820.929171:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443010816/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203820.929180:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203820.929184:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203820.929185:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203820.929189:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203820/real 1788203820] req@ffff90234598dc00 x1875064443010560/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203820.929192:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3092 00000100:00080000:2.0:1788203820.929200:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203820.929206:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203820.929210:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203820.929213:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443010560/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203820.929223:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443010944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203820.929229:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203820.929237:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203820.929243:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443010560:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203820.929266:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443010688:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203820.929283:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203820.929291:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203820/real 1788203820] req@ffff90234598c700 x1875064443010688/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203820.929299:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443010688/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203820.929304:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203820.929306:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203820.929308:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443010688:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203820.929312:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443010816:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203820.929322:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203820.929328:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203820/real 1788203820] req@ffff90234598ce00 x1875064443010816/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203820.929334:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443010816/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203820.929339:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203820.929341:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203820.929342:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443010816:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203820.929345:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443010944:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203820.929354:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203820.929358:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203820/real 1788203820] req@ffff90234598f480 x1875064443010944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203820.929365:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443010944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203820.929369:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203820.929370:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203820.929372:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443010944:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203821.761055:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203822.785033:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203823.809053:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203824.833027:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203825.857094:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203826.048950:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203826.048965:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203826.048970:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203826.048977:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3097 00000100:00080000:1.0:1788203826.048986:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203826.048991:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203826.049020:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064443011072/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203826.049039:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203826.049041:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203826.049042:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203826.049045:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3097 00000100:00080000:1.0:1788203826.049047:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203826.049049:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203826.049057:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203826.049066:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443011200/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203826.049075:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443011072:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203826.049076:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203826.049082:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203826.049085:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203826.049088:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3097 00000100:00080000:1.0:1788203826.049089:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203826.049091:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203826.049095:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203826.049102:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443011328/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203826.049107:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203826.049110:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203826.049111:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203826.049114:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3097 00000100:00080000:1.0:1788203826.049116:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203826.049118:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203826.049121:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203826.049128:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443011456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203826.049138:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203826.049178:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203826/real 1788203826] req@ffff90234519dc00 x1875064443011072/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203826.049197:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064443011072/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203826.049207:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203826.049211:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203826.049220:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443011072:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203826.049240:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443011200:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203826.049260:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203826.049272:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203826/real 1788203826] req@ffff90234519f480 x1875064443011200/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203826.049284:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443011200/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203826.049292:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203826.049296:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203826.049298:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443011200:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203826.049302:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443011328:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203826.049323:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203826.049330:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203826/real 1788203826] req@ffff90234519c380 x1875064443011328/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203826.049337:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443011328/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203826.049346:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203826.049348:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203826.049350:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443011328:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203826.049353:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443011456:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203826.049361:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203826.049367:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203826/real 1788203826] req@ffff90234519f100 x1875064443011456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203826.049373:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443011456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203826.049377:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203826.049379:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203826.049380:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443011456:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203826.881039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203827.905036:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203828.928987:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203829.953023:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203830.977035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203831.168971:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203831.168981:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203831.168984:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203831.168993:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3102 00000100:00080000:0.0:1788203831.169001:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203831.169006:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203831.169037:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443011584/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203831.169057:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203831.169059:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203831.169061:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203831.169063:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3102 00000100:00080000:0.0:1788203831.169065:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203831.169067:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203831.169073:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203831.169082:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443011712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203831.169088:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203831.169090:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203831.169092:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203831.169094:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3102 00000100:00080000:0.0:1788203831.169096:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203831.169098:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203831.169101:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203831.169108:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443011840/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203831.169108:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443011584:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203831.169122:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203831.169124:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203831.169125:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203831.169127:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3102 00000100:00080000:0.0:1788203831.169129:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203831.169131:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203831.169133:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203831.169140:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443011968/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203831.169173:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203831.169223:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203831/real 1788203831] req@ffff9023450b9500 x1875064443011584/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203831.169242:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443011584/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203831.169253:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203831.169257:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203831.169259:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443011584:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203831.169265:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443011712:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203831.169281:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203831.169290:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203831/real 1788203831] req@ffff9023450b8380 x1875064443011712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203831.169297:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443011712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203831.169302:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203831.169304:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203831.169306:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443011712:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203831.169321:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443011840:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203831.169331:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203831.169336:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203831/real 1788203831] req@ffff9023450b8a80 x1875064443011840/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203831.169343:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443011840/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203831.169347:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203831.169349:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203831.169350:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443011840:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203831.169353:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443011968:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203831.169361:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203831.169366:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203831/real 1788203831] req@ffff9023450bb800 x1875064443011968/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203831.169373:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443011968/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203831.169377:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203831.169379:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203831.169380:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443011968:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203832.001021:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203833.025057:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203834.049071:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203835.073049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203836.097022:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203836.288980:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203836.288989:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203836.288993:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203836.289002:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3107 00000100:00080000:0.0:1788203836.289012:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203836.289017:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203836.289057:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443012096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203836.289086:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203836.289089:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203836.289091:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203836.289095:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3107 00000100:00080000:0.0:1788203836.289104:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203836.289106:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203836.289114:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203836.289124:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443012224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203836.289133:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788203836.289134:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443012096:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203836.289136:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203836.289139:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203836.289150:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3107 00000100:00080000:0.0:1788203836.289153:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203836.289155:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203836.289158:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203836.289166:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443012352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203836.289172:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203836.289174:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203836.289176:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203836.289177:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3107 00000100:00080000:0.0:1788203836.289179:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203836.289181:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203836.289184:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203836.289190:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443012480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203836.289229:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443012224:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203836.289251:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443012352:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203836.289273:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443012480:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203837.121032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203838.144991:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203839.169024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788203839.361213:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788203839.361263:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788203839.361284:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788203839.361334:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788203839.361346:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788203839.361355:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788203839.361434:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203836/real 1788203839] req@ffff9023450b9c00 x1875064443012224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203839.361475:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443012224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203839.361488:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203839.361495:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203839.361500:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443012224:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203839.361508:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203836/real 1788203839] req@ffff9023450b9f80 x1875064443012352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203839.361519:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443012352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203839.361527:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203839.361530:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203839.361533:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443012352:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203839.361540:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203836/real 1788203839] req@ffff9023450b8700 x1875064443012480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203839.361550:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443012480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203839.361557:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203839.361561:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203839.361564:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443012480:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203839.361571:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203836/real 1788203839] req@ffff9023450bad80 x1875064443012096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203839.361584:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443012096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203839.361591:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203839.361595:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203839.361598:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443012096:192.168.201.130@tcp:250:kworker.0 00010000:00010000:2.0:1788203840.192981:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203841.217144:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203841.409120:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203841.409131:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203841.409137:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203841.409151:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3112 00000100:00080000:0.0:1788203841.409165:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203841.409174:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203841.409223:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443012608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203841.409269:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203841.409274:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203841.409276:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203841.409281:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3112 00000100:00080000:0.0:1788203841.409285:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203841.409291:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203841.409304:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203841.409326:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443012608:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203841.409327:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443012736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203841.409344:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203841.409348:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203841.409350:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203841.409355:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3112 00000100:00080000:0.0:1788203841.409361:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203841.409366:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203841.409373:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203841.409386:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443012864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203841.409400:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203841.409404:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203841.409406:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203841.409409:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3112 00000100:00080000:0.0:1788203841.409413:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203841.409417:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:2.0:1788203841.409422:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:0.0:1788203841.409423:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203841.409437:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443012992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203841.409476:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203841/real 1788203841] req@ffff9023450ba300 x1875064443012608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203841.409498:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443012608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203841.409509:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203841.409513:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203841.409517:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443012608:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203841.409547:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443012736:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203841.409570:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203841.409585:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203841/real 1788203841] req@ffff9023450b9180 x1875064443012736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203841.409595:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443012736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203841.409602:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203841.409605:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203841.409609:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443012736:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203841.409615:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443012864:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203841.409637:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203841.409647:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203841/real 1788203841] req@ffff9023450b8000 x1875064443012864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203841.409658:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443012864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203841.409666:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203841.409669:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203841.409672:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443012864:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203841.409678:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443012992:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203841.409709:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203841.409719:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203841/real 1788203841] req@ffff9023450baa00 x1875064443012992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203841.409730:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443012992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203841.409739:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203841.409743:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203841.409746:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443012992:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203842.241008:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203843.265040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203844.289067:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203845.313093:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203846.337040:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203846.529007:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203846.529016:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203846.529020:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203846.529031:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3118 00000100:00080000:2.0:1788203846.529042:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203846.529049:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203846.529089:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443013120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203846.529125:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203846.529129:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203846.529131:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203846.529136:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3118 00000100:00080000:2.0:1788203846.529140:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203846.529143:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203846.529154:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203846.529166:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443013248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203846.529174:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203846.529177:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203846.529178:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203846.529182:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3118 00000100:00080000:2.0:1788203846.529184:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203846.529187:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203846.529191:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203846.529204:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443013376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203846.529212:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443013120:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203846.529217:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203846.529231:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203846.529234:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203846.529240:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3118 00000100:00080000:2.0:1788203846.529243:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203846.529246:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203846.529252:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203846.529267:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443013504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203846.529299:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203846.529343:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203846/real 1788203846] req@ffff90234598c380 x1875064443013120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203846.529368:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443013120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203846.529379:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203846.529383:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203846.529387:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443013120:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203846.529396:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443013248:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203846.529418:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203846.529431:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203846/real 1788203846] req@ffff90234598d180 x1875064443013248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203846.529443:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443013248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203846.529450:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203846.529454:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203846.529456:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443013248:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203846.529479:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443013376:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203846.529498:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203846.529508:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203846/real 1788203846] req@ffff90234598e300 x1875064443013376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203846.529518:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443013376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203846.529530:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203846.529533:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203846.529537:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443013376:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203846.529542:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443013504:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203846.529557:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203846.529566:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203846/real 1788203846] req@ffff90234598df80 x1875064443013504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203846.529576:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443013504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203846.529582:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203846.529585:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203846.529589:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443013504:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203847.360998:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203848.385003:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203849.409668:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203850.433012:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203851.457093:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203851.648969:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203851.648977:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203851.648980:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203851.648987:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3123 00000100:00080000:2.0:1788203851.648996:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203851.649002:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203851.649031:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064443013632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203851.649049:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203851.649052:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203851.649053:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203851.649059:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3123 00000100:00080000:2.0:1788203851.649061:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203851.649063:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203851.649071:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203851.649080:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064443013760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203851.649092:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203851.649094:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203851.649096:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203851.649099:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443013632:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203851.649100:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3123 00000100:00080000:2.0:1788203851.649104:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203851.649109:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203851.649113:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203851.649124:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064443013888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203851.649132:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203851.649136:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203851.649138:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203851.649141:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3123 00000100:00080000:2.0:1788203851.649153:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203851.649157:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203851.649162:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203851.649166:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203851.649178:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443014016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203851.649200:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203851/real 1788203851] req@ffff90234598fb80 x1875064443013632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203851.649213:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064443013632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203851.649220:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203851.649222:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203851.649225:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443013632:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203851.649242:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443013760:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203851.649255:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203851.649262:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203851/real 1788203851] req@ffff90234598ed80 x1875064443013760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203851.649269:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064443013760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203851.649274:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203851.649276:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203851.649278:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443013760:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203851.649281:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443013888:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203851.649290:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203851.649295:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203851/real 1788203851] req@ffff90234598f100 x1875064443013888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203851.649301:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064443013888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203851.649306:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203851.649307:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203851.649309:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443013888:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203851.649312:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443014016:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203851.649321:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203851.649325:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203851/real 1788203851] req@ffff90234598df80 x1875064443014016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203851.649332:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443014016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203851.649336:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203851.649338:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203851.649347:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443014016:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203852.480995:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203853.505020:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203854.528994:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203855.553035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203856.577009:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203856.768977:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203856.768987:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203856.768991:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203856.769000:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3128 00000100:00080000:1.0:1788203856.769011:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203856.769018:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203856.769053:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443014144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203856.769074:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203856.769077:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203856.769078:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203856.769080:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3128 00000100:00080000:1.0:1788203856.769083:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203856.769086:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203856.769093:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203856.769101:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443014272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203856.769107:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203856.769109:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203856.769110:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203856.769112:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3128 00000100:00080000:1.0:1788203856.769114:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203856.769116:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203856.769119:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203856.769122:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443014144:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203856.769127:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443014400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203856.769135:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203856.769137:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203856.769137:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203856.769139:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3128 00000100:00080000:1.0:1788203856.769141:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203856.769143:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203856.769145:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203856.769151:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443014528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203856.769184:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203856.769221:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203856/real 1788203856] req@ffff90234519ea00 x1875064443014144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203856.769238:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443014144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203856.769248:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203856.769251:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203856.769255:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443014144:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203856.769262:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443014272:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203856.769282:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203856.769293:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203856/real 1788203856] req@ffff90234519c700 x1875064443014272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203856.769304:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443014272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203856.769317:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203856.769320:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203856.769323:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443014272:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203856.769357:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443014400:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203856.769375:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203856.769385:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203856/real 1788203856] req@ffff90234519fb80 x1875064443014400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203856.769396:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443014400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203856.769404:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203856.769407:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203856.769410:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443014400:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203856.769415:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443014528:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203856.769463:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203856.769473:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203856/real 1788203856] req@ffff90234519df80 x1875064443014528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203856.769484:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443014528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203856.769491:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203856.769494:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203856.769497:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443014528:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203857.601060:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203858.625014:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203859.649030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203860.672983:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203861.697125:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203861.888954:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203861.888962:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203861.888965:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203861.888972:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3133 00000100:00080000:2.0:1788203861.888981:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203861.888986:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203861.889016:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443014656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203861.889036:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203861.889038:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203861.889040:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203861.889042:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3133 00000100:00080000:2.0:1788203861.889044:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203861.889046:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203861.889053:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203861.889061:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443014784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203861.889067:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203861.889069:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203861.889071:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203861.889073:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3133 00000100:00080000:2.0:1788203861.889074:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203861.889079:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203861.889082:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203861.889082:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443014656:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203861.889090:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443014912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203861.889096:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203861.889098:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203861.889099:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203861.889101:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3133 00000100:00080000:2.0:1788203861.889104:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203861.889105:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203861.889108:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203861.889114:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443015040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203861.889145:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203861.889182:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203861/real 1788203861] req@ffff90234598e300 x1875064443014656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203861.889195:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443014656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203861.889202:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203861.889205:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203861.889208:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443014656:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203861.889213:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443014784:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203861.889227:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203861.889235:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203861/real 1788203861] req@ffff90234598d180 x1875064443014784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203861.889249:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443014784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203861.889257:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203861.889258:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203861.889261:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443014784:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203861.889281:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443014912:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203861.889300:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203861.889312:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203861/real 1788203861] req@ffff90234598c380 x1875064443014912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203861.889322:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443014912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203861.889330:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203861.889333:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203861.889349:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443014912:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203861.889355:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443015040:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203861.889374:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203861.889385:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203861/real 1788203861] req@ffff90234598f480 x1875064443015040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203861.889396:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443015040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203861.889404:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203861.889407:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203861.889410:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443015040:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203862.721037:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203863.745001:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203864.769035:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203865.793008:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203866.816999:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203867.008964:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203867.008972:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203867.008975:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203867.008982:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3138 00000100:00080000:2.0:1788203867.008991:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203867.008997:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203867.009028:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443015168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203867.009049:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203867.009052:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203867.009053:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203867.009055:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3138 00000100:00080000:2.0:1788203867.009057:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203867.009059:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203867.009065:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203867.009074:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443015296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203867.009081:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203867.009083:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203867.009085:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203867.009089:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3138 00000100:00100000:3.0:1788203867.009091:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443015168:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203867.009092:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203867.009098:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203867.009102:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203867.009108:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443015424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203867.009113:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203867.009116:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203867.009117:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203867.009119:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3138 00000100:00080000:2.0:1788203867.009121:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203867.009123:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203867.009126:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203867.009131:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443015552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203867.009145:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203867.009182:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203867/real 1788203867] req@ffff90234598ce00 x1875064443015168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203867.009196:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443015168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203867.009203:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203867.009205:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203867.009208:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443015168:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203867.009224:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443015296:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203867.009238:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203867.009245:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203867/real 1788203867] req@ffff90234598c700 x1875064443015296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203867.009267:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443015296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203867.009272:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203867.009274:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203867.009276:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443015296:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203867.009280:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443015424:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203867.009293:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203867.009299:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203867/real 1788203867] req@ffff90234598dc00 x1875064443015424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203867.009305:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443015424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203867.009310:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203867.009312:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203867.009313:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443015424:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203867.009317:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443015552:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203867.009325:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203867.009330:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203867/real 1788203867] req@ffff90234598f800 x1875064443015552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203867.009336:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443015552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203867.009340:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203867.009342:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203867.009343:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443015552:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203867.840993:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203868.865007:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203869.889031:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203870.913038:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203871.937042:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203872.129004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203872.129019:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203872.129025:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203872.129039:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3143 00000100:00080000:0.0:1788203872.129052:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203872.129060:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203872.129109:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443015680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203872.129144:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203872.129149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203872.129151:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203872.129155:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3143 00000100:00080000:0.0:1788203872.129172:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203872.129177:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203872.129189:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203872.129206:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443015808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203872.129216:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443015680:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203872.129217:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203872.129224:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203872.129227:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203872.129230:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3143 00000100:00080000:0.0:1788203872.129233:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203872.129236:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203872.129249:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203872.129260:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443015936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203872.129272:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203872.129275:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203872.129277:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203872.129284:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3143 00000800:00000100:3.0:1788203872.129289:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788203872.129290:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203872.129295:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203872.129299:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203872.129308:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443016064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203872.129342:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203872/real 1788203872] req@ffff9023450baa00 x1875064443015680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203872.129364:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443015680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203872.129376:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203872.129380:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203872.129383:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443015680:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203872.129406:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443015808:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203872.129425:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203872.129436:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203872/real 1788203872] req@ffff9023450b8000 x1875064443015808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203872.129445:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443015808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203872.129451:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203872.129453:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203872.129455:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443015808:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203872.129459:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443015936:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203872.129471:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203872.129478:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203872/real 1788203872] req@ffff9023450b9180 x1875064443015936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203872.129491:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443015936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203872.129499:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203872.129503:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203872.129506:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443015936:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203872.129513:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443016064:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203872.129544:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203872.129553:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203872/real 1788203872] req@ffff9023450ba300 x1875064443016064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203872.129561:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443016064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203872.129571:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203872.129573:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203872.129576:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443016064:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203872.961023:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203873.985035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203875.009023:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203876.033003:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203877.057021:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203877.248968:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203877.248977:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203877.248982:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203877.248991:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3148 00000100:00080000:0.0:1788203877.248999:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203877.249004:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203877.249032:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443016192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203877.249054:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203877.249056:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203877.249057:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203877.249061:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3148 00000100:00080000:0.0:1788203877.249064:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203877.249066:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203877.249083:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203877.249092:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443016192:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788203877.249097:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443016320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203877.249105:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203877.249108:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203877.249109:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203877.249112:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3148 00000100:00080000:0.0:1788203877.249115:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203877.249118:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203877.249122:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203877.249132:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443016448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203877.249138:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203877.249140:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203877.249142:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203877.249144:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3148 00000100:00080000:0.0:1788203877.249147:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788203877.249150:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788203877.249154:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203877.249157:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203877.249166:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443016576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203877.249186:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203877/real 1788203877] req@ffff9023450bad80 x1875064443016192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203877.249204:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443016192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203877.249211:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203877.249214:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203877.249217:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443016192:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203877.249233:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443016320:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203877.249247:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203877.249253:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203877/real 1788203877] req@ffff9023450b8700 x1875064443016320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203877.249261:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443016320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203877.249265:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203877.249267:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203877.249269:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443016320:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203877.249272:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443016448:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203877.249281:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203877.249287:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203877/real 1788203877] req@ffff9023450b9f80 x1875064443016448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203877.249296:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443016448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203877.249308:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203877.249312:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203877.249315:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443016448:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203877.249320:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443016576:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203877.249337:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203877.249348:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203877/real 1788203877] req@ffff9023450b9c00 x1875064443016576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203877.249355:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443016576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203877.249360:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203877.249362:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203877.249364:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443016576:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203878.081065:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203879.105101:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203880.128977:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203881.153044:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203882.177012:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203882.368952:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203882.368959:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203882.368962:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203882.368970:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3153 00000100:00080000:2.0:1788203882.368979:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203882.369002:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203882.369033:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064443016704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203882.369055:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203882.369058:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203882.369059:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203882.369062:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3153 00000100:00080000:2.0:1788203882.369064:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203882.369067:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203882.369074:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203882.369082:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064443016832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203882.369089:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203882.369093:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203882.369095:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203882.369095:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443016704:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203882.369102:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3153 00000100:00080000:2.0:1788203882.369104:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203882.369106:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203882.369108:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203882.369114:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064443016960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203882.369120:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203882.369122:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203882.369124:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203882.369126:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3153 00000100:00080000:2.0:1788203882.369127:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203882.369129:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203882.369133:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203882.369140:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443017088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203882.369169:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203882.369214:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203882/real 1788203882] req@ffff90234598d880 x1875064443016704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203882.369227:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064443016704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203882.369234:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203882.369236:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203882.369239:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443016704:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203882.369256:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443016832:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203882.369271:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203882.369279:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203882/real 1788203882] req@ffff90234598e680 x1875064443016832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203882.369286:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064443016832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203882.369291:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203882.369293:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203882.369295:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443016832:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203882.369300:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443016960:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203882.369317:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203882.369328:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203882/real 1788203882] req@ffff90234598ca80 x1875064443016960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203882.369340:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064443016960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203882.369348:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203882.369350:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203882.369352:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443016960:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203882.369356:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443017088:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203882.369369:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203882.369375:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203882/real 1788203882] req@ffff90234598f800 x1875064443017088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203882.369382:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443017088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203882.369386:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203882.369388:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203882.369390:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443017088:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203883.201032:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203884.225026:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203885.248994:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203886.273068:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203887.297088:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203887.489044:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203887.489055:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203887.489060:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203887.489070:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3159 00000100:00080000:0.0:1788203887.489082:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203887.489089:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203887.489139:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443017216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203887.489171:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203887.489174:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203887.489176:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203887.489179:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3159 00000100:00080000:0.0:1788203887.489183:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203887.489187:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203887.489198:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203887.489215:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443017344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203887.489230:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788203887.489233:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443017216:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203887.489235:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203887.489241:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203887.489251:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3159 00000100:00080000:0.0:1788203887.489256:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203887.489260:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203887.489265:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203887.489278:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443017472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203887.489286:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203887.489289:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203887.489291:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203887.489294:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3159 00000100:00080000:0.0:1788203887.489297:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203887.489299:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203887.489302:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203887.489312:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443017600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203887.489326:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203887.489375:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203887/real 1788203887] req@ffff9023450bb800 x1875064443017216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203887.489391:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443017216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203887.489400:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203887.489403:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203887.489406:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443017216:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203887.489426:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443017344:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203887.489442:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203887.489450:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203887/real 1788203887] req@ffff9023450b8a80 x1875064443017344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203887.489457:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443017344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203887.489462:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203887.489464:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203887.489466:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443017344:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203887.489470:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443017472:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203887.489487:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203887.489493:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203887/real 1788203887] req@ffff9023450b8380 x1875064443017472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203887.489500:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443017472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203887.489508:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203887.489510:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203887.489512:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443017472:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203887.489516:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443017600:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203887.489525:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203887.489530:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203887/real 1788203887] req@ffff9023450b9500 x1875064443017600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203887.489536:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443017600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203887.489541:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203887.489543:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203887.489544:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443017600:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203888.321010:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203889.345025:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203890.369038:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203891.393017:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203892.417031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203892.609016:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203892.609025:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203892.609029:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203892.609037:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3164 00000100:00080000:1.0:1788203892.609046:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203892.609052:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203892.609090:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443017728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203892.609119:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203892.609123:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203892.609125:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203892.609130:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3164 00000100:00080000:1.0:1788203892.609133:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203892.609138:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203892.609147:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203892.609162:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443017856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203892.609171:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203892.609173:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203892.609175:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203892.609178:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3164 00000100:00080000:1.0:1788203892.609181:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203892.609185:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203892.609190:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203892.609200:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443017984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203892.609207:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443017728:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203892.609213:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203892.609217:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203892.609218:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203892.609222:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3164 00000100:00080000:1.0:1788203892.609224:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203892.609226:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203892.609229:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203892.609245:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443018112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203892.609298:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203892.609344:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203892/real 1788203892] req@ffff90234519c000 x1875064443017728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203892.609361:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443017728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203892.609369:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203892.609373:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203892.609377:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443017728:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203892.609384:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443017856:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203892.609399:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203892.609408:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203892/real 1788203892] req@ffff90234519ed80 x1875064443017856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203892.609417:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443017856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203892.609422:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203892.609424:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203892.609426:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443017856:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203892.609450:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443017984:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203892.609463:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203892.609469:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203892/real 1788203892] req@ffff90234519d880 x1875064443017984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203892.609477:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443017984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203892.609482:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203892.609484:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203892.609486:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443017984:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203892.609490:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443018112:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203892.609500:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203892.609506:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203892/real 1788203892] req@ffff90234519e300 x1875064443018112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203892.609514:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443018112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203892.609519:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203892.609521:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203892.609523:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443018112:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203893.441103:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203894.465060:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203895.489029:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203896.513059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203897.537035:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203897.729063:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203897.729078:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203897.729084:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203897.729097:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3169 00000100:00080000:3.0:1788203897.729113:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203897.729122:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788203897.729169:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443018240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203897.729199:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203897.729202:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203897.729204:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203897.729207:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3169 00000100:00080000:3.0:1788203897.729210:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203897.729213:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203897.729223:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203897.729235:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064443018368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203897.729249:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203897.729252:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203897.729254:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203897.729256:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3169 00000100:00080000:3.0:1788203897.729258:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203897.729261:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203897.729265:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203897.729281:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443018496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203897.729291:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203897.729295:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203897.729299:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203897.729304:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3169 00000100:00100000:2.0:1788203897.729305:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443018240:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788203897.729310:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203897.729323:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788203897.729331:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203897.729355:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443018624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203897.729412:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203897.729458:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203897/real 1788203897] req@ffff902352688000 x1875064443018240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203897.729475:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443018240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203897.729501:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203897.729504:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203897.729507:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443018240:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203897.729513:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443018368:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203897.729532:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203897.729540:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203897/real 1788203897] req@ffff90235268b100 x1875064443018368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203897.729549:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064443018368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203897.729555:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203897.729557:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203897.729560:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443018368:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203897.729580:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443018496:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203897.729592:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203897.729598:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203897/real 1788203897] req@ffff90235268a300 x1875064443018496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203897.729606:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443018496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203897.729612:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203897.729614:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203897.729616:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443018496:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203897.729620:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443018624:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203897.729629:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203897.729635:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203897/real 1788203897] req@ffff90235268b800 x1875064443018624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203897.729642:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443018624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203897.729647:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203897.729649:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203897.729651:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443018624:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203898.561073:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203899.585074:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203900.609073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203901.633061:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203902.657085:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203902.848992:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203902.849001:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203902.849011:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203902.849022:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3174 00000100:00080000:0.0:1788203902.849032:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203902.849037:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203902.849077:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443018752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203902.849099:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203902.849102:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203902.849103:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203902.849107:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3174 00000100:00080000:0.0:1788203902.849111:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203902.849115:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203902.849125:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203902.849139:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064443018880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203902.849151:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203902.849155:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203902.849157:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203902.849162:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3174 00000100:00080000:0.0:1788203902.849166:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203902.849170:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443018752:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203902.849173:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203902.849182:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203902.849192:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064443019008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203902.849205:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203902.849208:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203902.849209:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203902.849211:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3174 00000100:00080000:0.0:1788203902.849215:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203902.849217:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203902.849220:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203902.849227:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064443019136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203902.849269:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443018880:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203902.849301:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443019008:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203902.849314:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443019136:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203903.681120:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203904.704973:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203905.728983:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788203905.921107:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:0.0:1788203905.921150:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788203905.921168:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788203905.921208:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788203905.921222:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788203905.921228:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203902/real 1788203905] req@ffff9023450bb480 x1875064443018752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1788203905.921240:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788203905.921281:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443018752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203905.921294:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203905.921300:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203905.921305:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443018752:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203905.921314:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203902/real 1788203905] req@ffff9023450ba680 x1875064443018880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203905.921329:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064443018880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203905.921337:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203905.921341:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203905.921344:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443018880:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203905.921375:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203902/real 1788203905] req@ffff9023450b8e00 x1875064443019008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203905.921388:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064443019008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203905.921396:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203905.921400:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203905.921403:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443019008:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203905.921410:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203902/real 1788203905] req@ffff9023450bbb80 x1875064443019136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203905.921420:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064443019136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203905.921428:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203905.921431:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203905.921434:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443019136:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203906.753035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203907.777018:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203907.969062:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203907.969076:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203907.969083:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203907.969095:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3179 00000100:00080000:1.0:1788203907.969107:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203907.969115:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203907.969168:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064443019264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203907.969207:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203907.969212:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203907.969214:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203907.969219:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3179 00000100:00080000:1.0:1788203907.969223:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203907.969229:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203907.969240:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203907.969260:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443019392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203907.969285:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203907.969290:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203907.969291:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203907.969296:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3179 00000100:00080000:1.0:1788203907.969300:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203907.969304:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203907.969310:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203907.969323:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443019520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203907.969333:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203907.969337:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203907.969339:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203907.969343:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3179 00000100:00080000:1.0:1788203907.969346:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203907.969350:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203907.969356:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203907.969367:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443019648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203907.969436:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443019264:192.168.201.130@tcp:250:kworker.0 00000800:00000100:3.0:1788203907.969525:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203907.969592:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203907/real 1788203907] req@ffff90234519f800 x1875064443019264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203907.969616:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064443019264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203907.969629:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203907.969634:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203907.969642:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443019264:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203907.969651:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443019392:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203907.969677:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203907.969691:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203907/real 1788203907] req@ffff90234519e300 x1875064443019392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203907.969703:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443019392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203907.969710:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203907.969713:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203907.969717:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443019392:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203907.969724:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443019520:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203907.969746:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203907.969757:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203907/real 1788203907] req@ffff90234519d880 x1875064443019520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203907.969769:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443019520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203907.969777:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203907.969781:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203907.969783:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443019520:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203907.969790:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443019648:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203907.969809:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203907.969819:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203907/real 1788203907] req@ffff90234519ed80 x1875064443019648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203907.969830:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443019648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203907.969971:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203907.969975:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203907.969979:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443019648:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203908.800991:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788203909.825058:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203910.849003:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203911.873082:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203912.897021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203913.088933:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203913.088941:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203913.088945:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203913.088954:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3184 00000100:00080000:1.0:1788203913.088963:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203913.088970:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203913.089000:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443019776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203913.089026:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203913.089030:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203913.089031:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203913.089034:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3184 00000100:00080000:1.0:1788203913.089037:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203913.089040:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203913.089049:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203913.089065:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443019904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203913.089074:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443019776:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203913.089081:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203913.089086:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203913.089089:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203913.089095:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3184 00000100:00080000:1.0:1788203913.089099:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203913.089103:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203913.089108:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203913.089120:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443020032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203913.089128:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203913.089131:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203913.089132:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203913.089135:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3184 00000100:00080000:1.0:1788203913.089138:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203913.089141:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203913.089145:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788203913.089150:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203913.089156:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443020160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203913.089190:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203913/real 1788203913] req@ffff90234519c000 x1875064443019776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203913.089212:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443019776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203913.089223:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203913.089228:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203913.089232:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443019776:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203913.089255:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443019904:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203913.089275:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203913.089285:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203913/real 1788203913] req@ffff90234519df80 x1875064443019904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203913.089294:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443019904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203913.089299:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203913.089302:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203913.089304:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443019904:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203913.089309:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443020032:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203913.089320:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203913.089326:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203913/real 1788203913] req@ffff90234519fb80 x1875064443020032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203913.089336:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443020032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203913.089342:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203913.089344:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203913.089347:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443020032:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203913.089350:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443020160:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203913.089360:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203913.089366:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203913/real 1788203913] req@ffff90234519c700 x1875064443020160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203913.089375:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443020160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203913.089380:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203913.089382:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203913.089385:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443020160:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788203913.921030:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203914.944997:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203915.969006:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203916.993076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788203918.017039:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203918.208989:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203918.209002:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203918.209007:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203918.209018:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3189 00000100:00080000:1.0:1788203918.209033:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203918.209043:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203918.209098:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443020288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203918.209129:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203918.209132:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203918.209133:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203918.209138:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3189 00000100:00080000:1.0:1788203918.209141:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203918.209145:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203918.209155:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203918.209169:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443020416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203918.209178:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203918.209180:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203918.209182:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203918.209191:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3189 00000100:00100000:2.0:1788203918.209193:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443020288:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203918.209196:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203918.209203:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203918.209211:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203918.209228:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443020544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203918.209238:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203918.209241:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203918.209243:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203918.209246:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3189 00000100:00080000:1.0:1788203918.209250:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203918.209254:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203918.209257:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203918.209266:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443020672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203918.209286:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203918.209337:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203918/real 1788203918] req@ffff90234519ea00 x1875064443020288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203918.209387:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443020288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203918.209400:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203918.209405:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203918.209410:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443020288:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203918.209436:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443020416:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203918.209461:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203918.209476:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203918/real 1788203918] req@ffff90234519f100 x1875064443020416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203918.209489:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443020416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203918.209499:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203918.209502:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203918.209506:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443020416:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203918.209513:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443020544:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203918.209537:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203918.209550:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203918/real 1788203918] req@ffff90234519c380 x1875064443020544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203918.209571:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443020544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203918.209581:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203918.209584:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203918.209588:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443020544:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203918.209595:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443020672:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203918.209614:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203918.209625:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203918/real 1788203918] req@ffff90234519f480 x1875064443020672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203918.209647:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443020672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203918.209659:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203918.209662:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203918.209666:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443020672:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203919.041116:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203920.065033:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203921.089039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203922.113027:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203923.137053:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203923.328981:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203923.328990:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203923.328993:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203923.329001:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3194 00000100:00080000:1.0:1788203923.329017:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203923.329023:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203923.329057:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064443020800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203923.329081:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203923.329083:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203923.329085:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203923.329088:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3194 00000100:00080000:1.0:1788203923.329091:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203923.329093:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203923.329100:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203923.329110:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443020928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203923.329118:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203923.329120:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203923.329122:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203923.329124:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3194 00000100:00080000:1.0:1788203923.329127:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203923.329131:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443020800:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203923.329132:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203923.329139:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203923.329149:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443021056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203923.329159:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203923.329163:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203923.329165:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203923.329169:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3194 00000100:00080000:1.0:1788203923.329173:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203923.329178:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203923.329184:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788203923.329189:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203923.329202:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443021184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203923.329226:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203923/real 1788203923] req@ffff90234519dc00 x1875064443020800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203923.329242:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064443020800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203923.329254:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203923.329257:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203923.329260:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443020800:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203923.329266:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443020928:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203923.329301:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203923.329311:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203923/real 1788203923] req@ffff90234519f480 x1875064443020928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203923.329319:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443020928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203923.329325:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203923.329327:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203923.329329:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443020928:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203923.329352:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443021056:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203923.329365:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203923.329371:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203923/real 1788203923] req@ffff90234519c380 x1875064443021056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203923.329379:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443021056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203923.329384:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203923.329386:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203923.329388:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443021056:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203923.329392:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443021184:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203923.329401:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203923.329406:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203923/real 1788203923] req@ffff90234519f100 x1875064443021184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203923.329413:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443021184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203923.329418:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203923.329420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203923.329422:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443021184:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203924.161022:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203925.185031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203926.209002:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203927.233050:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203928.257036:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203928.448975:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203928.448984:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203928.448987:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203928.448995:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3199 00000100:00080000:0.0:1788203928.449003:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203928.449009:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203928.449039:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064443021312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203928.449066:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203928.449069:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203928.449070:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203928.449073:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3199 00000100:00080000:0.0:1788203928.449076:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203928.449078:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203928.449087:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203928.449097:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064443021440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203928.449105:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203928.449108:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203928.449109:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203928.449111:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3199 00000100:00080000:0.0:1788203928.449115:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203928.449117:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443021312:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203928.449121:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203928.449125:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203928.449134:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064443021568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203928.449141:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203928.449143:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203928.449145:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203928.449148:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3199 00000100:00080000:0.0:1788203928.449150:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203928.449152:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203928.449155:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203928.449164:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443021696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203928.449202:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203928.449240:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203928/real 1788203928] req@ffff9023450bbb80 x1875064443021312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203928.449255:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064443021312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203928.449264:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203928.449268:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203928.449271:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443021312:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203928.449291:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443021440:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203928.449307:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203928.449316:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203928/real 1788203928] req@ffff9023450b8e00 x1875064443021440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203928.449325:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064443021440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203928.449331:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203928.449333:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203928.449335:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443021440:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203928.449339:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443021568:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203928.449349:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203928.449366:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203928/real 1788203928] req@ffff9023450ba680 x1875064443021568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203928.449376:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064443021568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203928.449385:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203928.449389:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203928.449392:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443021568:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203928.449399:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443021696:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203928.449418:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203928.449430:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203928/real 1788203928] req@ffff9023450bb480 x1875064443021696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203928.449441:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443021696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203928.449455:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203928.449458:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203928.449462:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443021696:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203929.281034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203930.305036:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203931.329050:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203932.353013:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203933.377010:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203933.568985:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203933.568995:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203933.568999:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203933.569007:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3205 00000100:00080000:0.0:1788203933.569016:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203933.569022:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788203933.569055:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443021824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203933.569079:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203933.569081:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203933.569083:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203933.569085:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3205 00000100:00080000:0.0:1788203933.569088:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203933.569090:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203933.569111:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203933.569123:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443021952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203933.569131:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203933.569134:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203933.569136:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788203933.569141:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443021824:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788203933.569142:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3205 00000100:00080000:0.0:1788203933.569147:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203933.569151:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203933.569155:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203933.569163:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443022080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203933.569170:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203933.569172:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203933.569174:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203933.569176:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3205 00000100:00080000:0.0:1788203933.569179:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203933.569182:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788203933.569184:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203933.569191:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443022208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203933.569202:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203933.569236:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203933/real 1788203933] req@ffff9023450b9500 x1875064443021824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203933.569250:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443021824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203933.569258:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203933.569261:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203933.569264:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443021824:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203933.569281:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443021952:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203933.569295:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203933.569303:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203933/real 1788203933] req@ffff9023450b8380 x1875064443021952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203933.569311:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443021952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203933.569316:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203933.569318:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203933.569320:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443021952:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203933.569325:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443022080:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203933.569335:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203933.569342:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203933/real 1788203933] req@ffff9023450b8a80 x1875064443022080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203933.569349:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443022080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203933.569354:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203933.569356:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203933.569358:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443022080:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203933.569361:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443022208:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203933.569371:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203933.569376:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203933/real 1788203933] req@ffff9023450bb800 x1875064443022208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203933.569383:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443022208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203933.569387:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203933.569389:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203933.569391:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443022208:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203934.401020:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203935.425007:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203936.449084:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203937.473048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203938.496990:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203938.688985:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203938.688993:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203938.688997:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203938.689020:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3210 00000100:00080000:2.0:1788203938.689032:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203938.689044:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203938.689082:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443022336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203938.689115:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203938.689119:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203938.689121:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203938.689125:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3210 00000100:00080000:2.0:1788203938.689129:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203938.689133:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203938.689143:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203938.689157:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443022464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203938.689167:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203938.689171:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203938.689175:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203938.689176:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443022336:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788203938.689182:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3210 00000100:00080000:2.0:1788203938.689188:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203938.689192:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203938.689197:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203938.689209:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443022592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203938.689224:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203938.689228:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203938.689230:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203938.689233:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3210 00000100:00080000:2.0:1788203938.689237:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203938.689240:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203938.689245:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203938.689250:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203938.689260:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443022720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203938.689296:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203938/real 1788203938] req@ffff90234598dc00 x1875064443022336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203938.689318:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443022336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203938.689330:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203938.689335:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203938.689339:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443022336:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203938.689364:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443022464:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203938.689393:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203938.689407:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203938/real 1788203938] req@ffff90234598c700 x1875064443022464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203938.689420:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443022464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203938.689430:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203938.689434:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203938.689437:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443022464:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203938.689443:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443022592:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203938.689464:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203938.689475:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203938/real 1788203938] req@ffff90234598ce00 x1875064443022592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203938.689488:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443022592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203938.689503:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203938.689507:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203938.689511:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443022592:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203938.689517:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443022720:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203938.689537:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203938.689548:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203938/real 1788203938] req@ffff90234598f480 x1875064443022720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203938.689561:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443022720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203938.689569:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203938.689573:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203938.689577:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443022720:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203939.521065:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203940.545137:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203941.569081:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203942.593054:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203943.617025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203943.808990:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203943.808997:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203943.809000:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203943.809009:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3215 00000100:00080000:2.0:1788203943.809019:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203943.809026:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203943.809060:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443022848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203943.809099:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203943.809103:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203943.809105:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203943.809110:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3215 00000100:00080000:2.0:1788203943.809114:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203943.809119:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203943.809128:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203943.809142:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443022848:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203943.809143:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443022976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203943.809152:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203943.809160:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203943.809161:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203943.809164:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3215 00000100:00080000:2.0:1788203943.809167:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203943.809170:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203943.809173:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203943.809182:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443023104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203943.809189:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203943.809192:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203943.809193:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203943.809195:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3215 00000100:00080000:2.0:1788203943.809210:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203943.809213:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203943.809216:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203943.809221:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203943.809228:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443023232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203943.809259:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203943/real 1788203943] req@ffff90234598c380 x1875064443022848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203943.809273:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443022848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203943.809286:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203943.809290:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203943.809292:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443022848:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203943.809313:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443022976:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203943.809329:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203943.809338:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203943/real 1788203943] req@ffff90234598d180 x1875064443022976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203943.809347:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443022976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203943.809352:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203943.809354:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203943.809357:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443022976:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203943.809361:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443023104:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203943.809371:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203943.809377:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203943/real 1788203943] req@ffff90234598e300 x1875064443023104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203943.809385:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443023104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203943.809390:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203943.809392:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203943.809394:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443023104:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203943.809397:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443023232:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203943.809406:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203943.809412:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203943/real 1788203943] req@ffff90234598df80 x1875064443023232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203943.809420:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443023232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203943.809425:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203943.809427:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203943.809429:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443023232:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203944.641054:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203945.665042:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203946.689113:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203947.713106:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203948.737074:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203948.929021:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203948.929031:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203948.929036:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203948.929047:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3220 00000100:00080000:1.0:1788203948.929059:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203948.929068:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203948.929118:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443023360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203948.929142:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203948.929144:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203948.929146:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203948.929150:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3220 00000100:00080000:1.0:1788203948.929153:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203948.929157:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203948.929171:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203948.929181:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443023360:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203948.929187:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443023488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203948.929197:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203948.929200:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203948.929202:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203948.929205:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3220 00000100:00080000:1.0:1788203948.929208:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203948.929212:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203948.929216:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203948.929226:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443023616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203948.929234:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203948.929237:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203948.929242:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788203948.929242:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788203948.929251:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3220 00000100:00080000:1.0:1788203948.929256:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203948.929259:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203948.929270:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203948.929282:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443023744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203948.929284:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203948/real 1788203948] req@ffff90234519ea00 x1875064443023360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203948.929314:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443023360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203948.929327:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203948.929332:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203948.929336:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443023360:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203948.929360:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443023488:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203948.929380:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203948.929394:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203948/real 1788203948] req@ffff90234519c700 x1875064443023488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203948.929408:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443023488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203948.929427:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203948.929431:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203948.929435:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443023488:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203948.929441:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443023616:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203948.929458:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203948.929467:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203948/real 1788203948] req@ffff90234519fb80 x1875064443023616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203948.929475:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443023616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203948.929485:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203948.929488:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203948.929490:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443023616:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203948.929494:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443023744:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203948.929505:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203948.929513:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203948/real 1788203948] req@ffff90234519df80 x1875064443023744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203948.929522:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443023744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203948.929528:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203948.929530:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203948.929533:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443023744:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203949.761073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203950.785082:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203951.809085:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203952.833117:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203953.857031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203954.049018:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203954.049030:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203954.049034:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203954.049042:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3225 00000100:00080000:2.0:1788203954.049051:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203954.049057:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788203954.049095:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064443023872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203954.049134:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203954.049138:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203954.049139:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203954.049143:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3225 00000100:00080000:2.0:1788203954.049146:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203954.049149:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203954.049157:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203954.049170:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064443024000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203954.049177:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203954.049180:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203954.049181:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203954.049184:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3225 00000100:00080000:2.0:1788203954.049186:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203954.049188:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203954.049195:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203954.049205:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443023872:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203954.049209:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064443024128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203954.049226:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203954.049229:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203954.049230:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203954.049232:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3225 00000100:00080000:2.0:1788203954.049233:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203954.049235:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788203954.049239:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203954.049250:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443024256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203954.049300:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203954.049349:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203954/real 1788203954] req@ffff90234598f100 x1875064443023872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203954.049365:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064443023872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203954.049374:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203954.049383:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203954.049386:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443023872:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203954.049393:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443024000:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203954.049409:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203954.049418:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203954/real 1788203954] req@ffff90234598ed80 x1875064443024000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203954.049426:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064443024000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203954.049436:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203954.049439:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203954.049441:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443024000:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203954.049465:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443024128:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203954.049484:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203954.049491:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203954/real 1788203954] req@ffff90234598fb80 x1875064443024128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203954.049499:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064443024128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203954.049506:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203954.049508:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203954.049510:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443024128:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203954.049514:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443024256:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203954.049524:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203954.049530:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203954/real 1788203954] req@ffff90234598df80 x1875064443024256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203954.049538:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443024256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203954.049543:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203954.049544:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203954.049547:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443024256:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203954.881062:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203955.905079:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203956.929055:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203957.953082:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203958.977056:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203959.169039:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203959.169054:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203959.169059:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203959.169070:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3230 00000100:00080000:1.0:1788203959.169084:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203959.169091:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203959.169139:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443024384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203959.169167:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203959.169169:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203959.169171:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203959.169174:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3230 00000100:00080000:1.0:1788203959.169177:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203959.169179:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203959.169189:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203959.169199:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443024512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203959.169206:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203959.169209:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203959.169210:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203959.169212:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3230 00000100:00080000:1.0:1788203959.169214:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788203959.169219:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443024384:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203959.169220:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203959.169226:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203959.169233:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443024640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203959.169239:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203959.169242:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203959.169244:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203959.169247:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3230 00000100:00080000:1.0:1788203959.169250:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203959.169253:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203959.169256:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203959.169267:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443024768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203959.169303:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203959.169343:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203959/real 1788203959] req@ffff90234519c000 x1875064443024384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203959.169361:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443024384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203959.169370:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203959.169375:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203959.169379:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443024384:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203959.169401:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443024512:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203959.169416:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203959.169425:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203959/real 1788203959] req@ffff90234519ed80 x1875064443024512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203959.169434:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443024512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203959.169439:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203959.169441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203959.169443:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443024512:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203959.169447:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443024640:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203959.169458:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203959.169465:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203959/real 1788203959] req@ffff90234519d880 x1875064443024640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203959.169472:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443024640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203959.169478:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203959.169480:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203959.169481:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443024640:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203959.169485:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443024768:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203959.169494:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203959.169500:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203959/real 1788203959] req@ffff90234519e300 x1875064443024768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203959.169508:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443024768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203959.169513:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203959.169515:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203959.169516:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443024768:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203960.001090:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203961.025056:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203962.049096:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203963.073073:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203964.097012:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203964.288988:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203964.289000:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203964.289004:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203964.289014:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3235 00000100:00080000:1.0:1788203964.289024:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203964.289031:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203964.289065:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064443024896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203964.289090:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203964.289095:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203964.289097:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203964.289107:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3235 00000100:00080000:1.0:1788203964.289111:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203964.289115:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203964.289126:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203964.289141:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443025024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203964.289143:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443024896:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203964.289153:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203964.289156:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203964.289158:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203964.289162:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3235 00000100:00080000:1.0:1788203964.289165:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203964.289169:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203964.289173:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203964.289182:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443025152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203964.289191:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203964.289194:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203964.289196:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203964.289199:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3235 00000100:00080000:1.0:1788203964.289202:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203964.289205:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203964.289209:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203964.289210:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788203964.289218:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443025280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203964.289247:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203964/real 1788203964] req@ffff90234519f800 x1875064443024896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203964.289274:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064443024896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203964.289282:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203964.289285:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203964.289288:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443024896:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788203964.289307:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443025024:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788203964.289325:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203964.289334:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203964/real 1788203964] req@ffff90234519e300 x1875064443025024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203964.289342:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443025024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203964.289347:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203964.289350:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203964.289352:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443025024:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203964.289355:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443025152:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203964.289366:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203964.289372:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203964/real 1788203964] req@ffff90234519d880 x1875064443025152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203964.289379:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443025152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203964.289384:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203964.289386:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203964.289388:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443025152:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203964.289392:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443025280:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788203964.289401:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788203964.289407:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203964/real 1788203964] req@ffff90234519ed80 x1875064443025280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203964.289414:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443025280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203964.289419:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203964.289421:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788203964.289422:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443025280:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203965.121016:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203966.145035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203967.169046:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203968.193066:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203969.217055:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203969.408939:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203969.408948:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203969.408951:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203969.408960:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3240 00000100:00080000:1.0:1788203969.408969:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203969.408974:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203969.409007:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443025408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203969.409039:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203969.409043:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203969.409045:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203969.409050:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3240 00000100:00080000:1.0:1788203969.409054:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203969.409059:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203969.409069:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203969.409087:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443025408:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788203969.409089:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443025536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203969.409105:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203969.409110:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203969.409112:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203969.409117:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3240 00000100:00080000:1.0:1788203969.409122:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203969.409128:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203969.409134:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203969.409145:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443025664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203969.409154:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203969.409157:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203969.409158:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203969.409161:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3240 00000100:00080000:1.0:1788203969.409164:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203969.409167:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203969.409170:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203969.409178:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443025792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203969.409207:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443025536:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788203969.409236:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443025664:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788203969.409250:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443025792:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203970.241094:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203971.265136:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203972.288980:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788203972.481306:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:1.0:1788203972.481372:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788203972.481400:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788203972.481466:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788203972.481477:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788203972.481486:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:2.0:1788203972.481570:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203969/real 1788203972] req@ffff90234519df80 x1875064443025536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203972.481598:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443025536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203972.481610:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203972.481615:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203972.481619:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443025536:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203972.481627:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203969/real 1788203972] req@ffff90234519fb80 x1875064443025664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203972.481641:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443025664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203972.481650:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203972.481654:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203972.481658:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443025664:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203972.481665:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203969/real 1788203972] req@ffff90234519c700 x1875064443025792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203972.481678:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443025792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203972.481693:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203972.481697:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203972.481701:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443025792:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203972.481707:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203969/real 1788203972] req@ffff90234519c000 x1875064443025408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203972.481720:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443025408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203972.481728:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203972.481731:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203972.481735:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443025408:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788203973.313006:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203974.337038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203974.529013:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203974.529024:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203974.529029:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203974.529039:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3246 00000100:00080000:1.0:1788203974.529052:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203974.529060:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203974.529104:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443025920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203974.529146:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203974.529151:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203974.529153:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203974.529158:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3246 00000100:00080000:1.0:1788203974.529161:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203974.529166:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203974.529178:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203974.529196:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443026048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203974.529209:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203974.529216:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788203974.529216:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443025920:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203974.529218:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203974.529229:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3246 00000100:00080000:1.0:1788203974.529233:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203974.529236:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203974.529241:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203974.529254:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443026176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203974.529262:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203974.529266:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203974.529268:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203974.529272:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3246 00000100:00080000:1.0:1788203974.529277:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203974.529282:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203974.529287:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203974.529300:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443026304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203974.529316:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203974.529377:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203974/real 1788203974] req@ffff90234519ea00 x1875064443025920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203974.529418:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443025920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203974.529436:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203974.529441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203974.529446:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443025920:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203974.529495:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443026048:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203974.529526:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203974.529543:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203974/real 1788203974] req@ffff90234519f100 x1875064443026048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203974.529558:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443026048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203974.529567:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203974.529572:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203974.529576:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443026048:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203974.529583:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443026176:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203974.529606:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203974.529618:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203974/real 1788203974] req@ffff90234519c380 x1875064443026176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203974.529632:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443026176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203974.529642:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203974.529646:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203974.529650:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443026176:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203974.529657:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443026304:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203974.529677:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203974.529688:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203974/real 1788203974] req@ffff90234519f480 x1875064443026304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203974.529710:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443026304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203974.529719:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203974.529723:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203974.529727:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443026304:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203975.361041:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203976.385034:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203977.409004:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203978.433051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203979.457117:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203979.648979:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203979.648987:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203979.648991:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203979.649000:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3251 00000100:00080000:1.0:1788203979.649009:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203979.649014:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203979.649054:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064443026432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203979.649084:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203979.649088:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203979.649090:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203979.649093:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3251 00000100:00080000:1.0:1788203979.649096:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203979.649100:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203979.649108:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203979.649121:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443026560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203979.649135:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788203979.649137:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443026432:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203979.649138:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203979.649142:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203979.649149:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3251 00000100:00080000:1.0:1788203979.649151:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203979.649153:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203979.649157:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203979.649166:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443026688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203979.649173:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203979.649176:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203979.649177:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203979.649180:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3251 00000100:00080000:1.0:1788203979.649182:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203979.649184:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203979.649186:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203979.649194:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443026816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203979.649200:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203979.649238:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203979/real 1788203979] req@ffff90234519dc00 x1875064443026432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203979.649252:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064443026432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203979.649260:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203979.649263:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203979.649266:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443026432:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203979.649284:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443026560:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203979.649320:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203979.649329:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203979/real 1788203979] req@ffff90234519f480 x1875064443026560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203979.649337:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443026560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203979.649347:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203979.649349:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203979.649351:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443026560:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203979.649355:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443026688:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203979.649367:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203979.649374:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203979/real 1788203979] req@ffff90234519c380 x1875064443026688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203979.649381:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443026688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203979.649386:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203979.649388:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203979.649390:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443026688:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203979.649394:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443026816:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203979.649404:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203979.649409:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203979/real 1788203979] req@ffff90234519f100 x1875064443026816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203979.649416:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443026816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203979.649421:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203979.649423:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203979.649425:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443026816:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203980.481032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203981.505075:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203982.529047:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203983.553056:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203984.577012:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203984.768955:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203984.768963:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203984.768967:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203984.768976:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3256 00000100:00080000:1.0:1788203984.768985:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203984.768991:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203984.769033:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443026944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203984.769055:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203984.769058:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203984.769060:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203984.769064:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3256 00000100:00080000:1.0:1788203984.769066:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203984.769069:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203984.769077:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203984.769087:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443027072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203984.769096:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203984.769099:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203984.769101:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788203984.769104:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443026944:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203984.769106:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3256 00000100:00080000:1.0:1788203984.769113:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203984.769116:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203984.769119:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203984.769130:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443027200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203984.769142:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203984.769144:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203984.769146:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203984.769149:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3256 00000100:00080000:1.0:1788203984.769152:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203984.769156:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203984.769160:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203984.769170:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443027328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203984.769172:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203984.769281:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203984/real 1788203984] req@ffff90234519ea00 x1875064443026944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203984.769306:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443026944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203984.769318:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203984.769323:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203984.769329:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443026944:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203984.769355:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443027072:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203984.769381:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203984.769394:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203984/real 1788203984] req@ffff90234519c000 x1875064443027072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203984.769407:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443027072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203984.769415:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203984.769418:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203984.769422:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443027072:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203984.769428:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443027200:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203984.769449:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203984.769463:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203984/real 1788203984] req@ffff90234519c700 x1875064443027200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203984.769473:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443027200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203984.769479:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203984.769481:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203984.769483:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443027200:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203984.769487:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443027328:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203984.769503:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203984.769512:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203984/real 1788203984] req@ffff90234519fb80 x1875064443027328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203984.769520:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443027328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203984.769526:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203984.769528:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203984.769530:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443027328:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788203985.601030:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203986.625085:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203987.649084:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203988.673020:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203989.697050:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203989.888987:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203989.888996:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203989.889000:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203989.889012:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3261 00000100:00080000:1.0:1788203989.889023:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203989.889029:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203989.889060:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443027456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203989.889080:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203989.889083:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203989.889084:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203989.889088:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3261 00000100:00080000:1.0:1788203989.889091:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203989.889094:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203989.889102:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203989.889113:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443027584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203989.889137:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443027456:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203989.889148:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203989.889152:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203989.889153:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203989.889157:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3261 00000100:00080000:1.0:1788203989.889159:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203989.889162:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203989.889167:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203989.889177:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443027712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203989.889185:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203989.889187:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203989.889189:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203989.889191:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3261 00000100:00080000:1.0:1788203989.889194:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203989.889196:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203989.889198:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203989.889205:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443027840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203989.889214:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203989.889248:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203989/real 1788203989] req@ffff90234519df80 x1875064443027456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203989.889262:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443027456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203989.889270:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203989.889273:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203989.889276:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443027456:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203989.889301:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443027584:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203989.889326:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203989.889338:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203989/real 1788203989] req@ffff90234519ed80 x1875064443027584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203989.889348:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443027584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203989.889354:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203989.889356:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203989.889358:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443027584:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203989.889363:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443027712:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203989.889377:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203989.889384:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203989/real 1788203989] req@ffff90234519d880 x1875064443027712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203989.889392:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443027712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203989.889397:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203989.889399:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203989.889401:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443027712:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203989.889405:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443027840:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203989.889415:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203989.889421:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203989/real 1788203989] req@ffff90234519e300 x1875064443027840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203989.889428:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443027840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203989.889434:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203989.889436:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203989.889437:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443027840:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203990.721036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203991.745024:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203992.769017:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203993.793027:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203994.817001:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203995.008980:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203995.008990:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203995.008994:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203995.009002:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3266 00000100:00080000:1.0:1788203995.009012:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203995.009017:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788203995.009074:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064443027968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203995.009099:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203995.009102:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203995.009104:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203995.009107:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3266 00000100:00080000:1.0:1788203995.009110:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203995.009113:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203995.009121:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203995.009132:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443028096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203995.009141:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203995.009143:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203995.009150:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788203995.009150:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443027968:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788203995.009159:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3266 00000100:00080000:1.0:1788203995.009162:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203995.009165:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203995.009168:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203995.009177:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443028224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203995.009184:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203995.009187:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203995.009188:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203995.009191:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3266 00000100:00080000:1.0:1788203995.009193:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203995.009195:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788203995.009198:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203995.009206:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443028352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203995.009211:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203995.009247:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203995/real 1788203995] req@ffff90234519f800 x1875064443027968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203995.009261:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064443027968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203995.009269:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203995.009272:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203995.009275:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443027968:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788203995.009294:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443028096:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788203995.009318:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203995.009332:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203995/real 1788203995] req@ffff90234519e300 x1875064443028096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203995.009346:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443028096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203995.009356:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203995.009359:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203995.009362:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443028096:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788203995.009367:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443028224:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203995.009383:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203995.009392:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203995/real 1788203995] req@ffff90234519d880 x1875064443028224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203995.009399:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443028224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203995.009405:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203995.009407:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203995.009409:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443028224:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788203995.009412:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443028352:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788203995.009422:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788203995.009427:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203995/real 1788203995] req@ffff90234519ed80 x1875064443028352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203995.009435:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443028352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203995.009444:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203995.009446:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788203995.009448:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443028352:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788203995.841052:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203996.865052:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203997.889067:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203998.913078:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203999.938170:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204000.129043:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204000.129054:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204000.129058:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204000.129067:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3271 00000100:00080000:3.0:1788204000.129078:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204000.129085:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204000.129124:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443028480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204000.129169:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204000.129173:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204000.129175:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204000.129180:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3271 00000100:00080000:3.0:1788204000.129184:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204000.129188:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204000.129199:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204000.129213:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443028480:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204000.129216:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443028608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204000.129230:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204000.129234:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204000.129236:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204000.129240:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3271 00000100:00080000:3.0:1788204000.129244:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204000.129248:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204000.129253:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204000.129264:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443028736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204000.129272:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204000.129274:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204000.129278:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204000.129279:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204000.129284:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3271 00000100:00080000:3.0:1788204000.129289:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204000.129292:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204000.129296:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204000.129312:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204000/real 1788204000] req@ffff902352689500 x1875064443028480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204000.129312:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443028864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204000.129350:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443028480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204000.129359:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204000.129362:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204000.129365:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443028480:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204000.129383:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443028608:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204000.129398:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204000.129406:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204000/real 1788204000] req@ffff902352688e00 x1875064443028608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204000.129413:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443028608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204000.129419:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204000.129421:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204000.129423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443028608:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204000.129427:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443028736:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204000.129437:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204000.129443:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204000/real 1788204000] req@ffff90235268a680 x1875064443028736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204000.129451:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443028736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204000.129456:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204000.129458:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204000.129460:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443028736:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204000.129463:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443028864:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204000.129472:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204000.129477:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204000/real 1788204000] req@ffff902352689c00 x1875064443028864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204000.129484:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443028864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204000.129489:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204000.129491:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204000.129492:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443028864:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204000.961055:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204001.985008:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204003.009032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204004.033027:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204005.057075:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204005.249041:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204005.249052:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204005.249056:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204005.249067:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3276 00000100:00080000:0.0:1788204005.249080:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204005.249087:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204005.249132:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443028992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204005.249154:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204005.249156:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204005.249158:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204005.249160:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3276 00000100:00080000:0.0:1788204005.249162:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204005.249165:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204005.249174:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204005.249183:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443029120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204005.249190:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204005.249192:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204005.249193:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204005.249196:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3276 00000100:00080000:0.0:1788204005.249197:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204005.249199:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204005.249202:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204005.249211:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443029248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204005.249217:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443028992:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204005.249219:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204005.249226:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204005.249230:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204005.249236:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3276 00000100:00080000:0.0:1788204005.249239:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204005.249241:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204005.249245:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204005.249258:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443029376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204005.249299:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204005.249338:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204005/real 1788204005] req@ffff9023450b9c00 x1875064443028992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204005.249355:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443028992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204005.249364:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204005.249368:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204005.249372:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443028992:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204005.249380:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443029120:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204005.249398:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204005.249409:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204005/real 1788204005] req@ffff9023450b9f80 x1875064443029120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204005.249419:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443029120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204005.249427:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204005.249430:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204005.249433:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443029120:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204005.249454:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443029248:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204005.249469:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204005.249478:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204005/real 1788204005] req@ffff9023450b8700 x1875064443029248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204005.249488:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443029248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204005.249501:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204005.249504:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204005.249506:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443029248:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204005.249512:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443029376:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204005.249527:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204005.249535:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204005/real 1788204005] req@ffff9023450bad80 x1875064443029376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204005.249545:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443029376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204005.249551:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204005.249554:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204005.249557:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443029376:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204006.081101:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204007.105056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204008.129077:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204009.153066:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204010.177071:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204010.368937:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204010.368945:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204010.368948:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204010.368957:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3281 00000100:00080000:0.0:1788204010.368965:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204010.368971:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204010.369001:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443029504/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204010.369023:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204010.369026:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204010.369027:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204010.369029:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3281 00000100:00080000:0.0:1788204010.369031:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204010.369033:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204010.369040:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204010.369051:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443029632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204010.369062:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204010.369065:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204010.369069:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788204010.369070:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443029504:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204010.369075:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3281 00000100:00080000:0.0:1788204010.369080:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204010.369084:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204010.369090:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204010.369102:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443029760/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204010.369112:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204010.369116:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204010.369118:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204010.369121:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3281 00000100:00080000:0.0:1788204010.369125:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204010.369129:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:2.0:1788204010.369134:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:0.0:1788204010.369136:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204010.369149:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443029888/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204010.369175:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204010/real 1788204010] req@ffff9023450ba300 x1875064443029504/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204010.369188:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443029504/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204010.369196:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204010.369200:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204010.369205:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443029504:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204010.369228:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443029632:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204010.369250:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204010.369262:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204010/real 1788204010] req@ffff9023450b9180 x1875064443029632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204010.369274:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443029632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204010.369282:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204010.369285:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204010.369288:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443029632:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204010.369293:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443029760:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204010.369307:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204010.369314:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204010/real 1788204010] req@ffff9023450b8000 x1875064443029760/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204010.369321:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443029760/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204010.369326:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204010.369328:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204010.369330:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443029760:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204010.369332:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443029888:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204010.369341:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204010.369346:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204010/real 1788204010] req@ffff9023450baa00 x1875064443029888/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204010.369370:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443029888/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204010.369377:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204010.369380:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204010.369383:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443029888:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204011.201040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204012.225019:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204013.249025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204014.273013:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204015.297079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204015.489071:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204015.489086:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204015.489093:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204015.489110:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3287 00000100:00080000:1.0:1788204015.489128:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204015.489140:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204015.489199:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443030016/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204015.489241:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204015.489246:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204015.489248:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204015.489254:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3287 00000100:00080000:1.0:1788204015.489258:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204015.489265:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204015.489279:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204015.489292:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443030016:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204015.489304:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443030144/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204015.489319:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204015.489324:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204015.489326:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204015.489333:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3287 00000100:00080000:1.0:1788204015.489338:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204015.489342:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204015.489349:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204015.489368:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443030272/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204015.489376:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788204015.489398:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204015.489403:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204015.489405:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204015.489410:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3287 00000100:00080000:1.0:1788204015.489415:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204015.489422:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204015.489431:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204015.489435:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204015/real 1788204015] req@ffff90234519df80 x1875064443030016/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204015.489455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443030016/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204015.489460:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443030400/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204015.489465:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204015.489471:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204015.489476:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443030016:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204015.489501:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443030144:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204015.489517:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204015.489526:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204015/real 1788204015] req@ffff90234519fb80 x1875064443030144/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204015.489534:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443030144/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204015.489550:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204015.489553:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204015.489555:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443030144:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204015.489559:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443030272:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204015.489570:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204015.489576:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204015/real 1788204015] req@ffff90234519c700 x1875064443030272/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204015.489583:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443030272/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204015.489589:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204015.489591:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204015.489592:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443030272:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204015.489595:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443030400:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204015.489604:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204015.489609:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204015/real 1788204015] req@ffff90234519c000 x1875064443030400/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204015.489615:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443030400/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204015.489619:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204015.489621:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204015.489623:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443030400:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204016.321011:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204017.345018:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204018.369061:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204019.393011:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204020.417076:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204020.609034:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204020.609048:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204020.609054:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204020.609066:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3292 00000100:00080000:0.0:1788204020.609079:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204020.609090:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204020.609145:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443030528/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204020.609178:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204020.609182:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204020.609184:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204020.609190:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3292 00000100:00080000:0.0:1788204020.609194:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204020.609199:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204020.609209:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204020.609225:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443030656/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204020.609236:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204020.609242:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788204020.609242:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443030528:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204020.609245:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204020.609255:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3292 00000100:00080000:0.0:1788204020.609259:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204020.609264:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204020.609270:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204020.609285:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443030784/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204020.609295:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204020.609299:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204020.609301:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204020.609306:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3292 00000100:00080000:0.0:1788204020.609310:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204020.609314:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204020.609320:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788204020.609321:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204020.609335:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443030912/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204020.609362:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204020/real 1788204020] req@ffff9023450baa00 x1875064443030528/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204020.609385:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443030528/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204020.609396:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204020.609400:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204020.609404:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443030528:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204020.609429:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443030656:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204020.609446:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204020.609454:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204020/real 1788204020] req@ffff9023450b8000 x1875064443030656/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204020.609462:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443030656/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204020.609467:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204020.609469:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204020.609471:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443030656:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204020.609475:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443030784:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204020.609486:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204020.609492:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204020/real 1788204020] req@ffff9023450b9180 x1875064443030784/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204020.609498:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443030784/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204020.609503:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204020.609505:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204020.609506:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443030784:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204020.609510:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443030912:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204020.609519:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204020.609524:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204020/real 1788204020] req@ffff9023450ba300 x1875064443030912/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204020.609530:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443030912/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204020.609535:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204020.609537:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204020.609538:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443030912:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204021.441087:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204022.465035:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204023.489030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204024.513095:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204025.537029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204025.728974:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204025.728983:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204025.728986:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204025.728993:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3297 00000100:00080000:3.0:1788204025.729002:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204025.729008:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204025.729052:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443031040/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204025.729079:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204025.729082:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204025.729083:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204025.729086:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3297 00000100:00080000:3.0:1788204025.729090:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204025.729094:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204025.729101:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204025.729113:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443031040:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204025.729114:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443031168/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204025.729129:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204025.729134:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204025.729136:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204025.729140:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3297 00000100:00080000:3.0:1788204025.729144:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204025.729148:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204025.729154:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788204025.729172:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204025.729175:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443031296/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204025.729183:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204025.729185:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204025.729187:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204025.729191:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3297 00000100:00080000:3.0:1788204025.729195:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204025.729199:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204025.729204:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204025.729209:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204025/real 1788204025] req@ffff902352689180 x1875064443031040/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204025.729217:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443031424/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204025.729223:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443031040/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204025.729238:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204025.729240:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204025.729244:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443031040:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204025.729262:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443031168:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204025.729274:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204025.729282:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204025/real 1788204025] req@ffff902352688700 x1875064443031168/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204025.729289:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443031168/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204025.729293:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204025.729295:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204025.729297:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443031168:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204025.729300:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443031296:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204025.729309:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204025.729314:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204025/real 1788204025] req@ffff90235268bb80 x1875064443031296/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204025.729320:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443031296/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204025.729325:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204025.729326:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204025.729328:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443031296:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204025.729331:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443031424:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204025.729339:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204025.729344:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204025/real 1788204025] req@ffff902352689880 x1875064443031424/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204025.729350:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443031424/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204025.729354:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204025.729356:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204025.729357:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443031424:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204026.561102:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204027.585080:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204028.609081:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204029.633052:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204030.657082:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204030.849007:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204030.849017:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204030.849021:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204030.849032:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3302 00000100:00080000:3.0:1788204030.849043:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204030.849050:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204030.849098:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443031552/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204030.849123:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204030.849126:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204030.849130:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204030.849133:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3302 00000100:00080000:3.0:1788204030.849135:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204030.849138:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204030.849149:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204030.849159:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443031680/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204030.849165:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204030.849168:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204030.849169:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204030.849173:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3302 00000100:00100000:2.0:1788204030.849174:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443031552:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204030.849175:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204030.849180:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204030.849184:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204030.849192:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443031808/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204030.849198:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204030.849200:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204030.849202:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204030.849204:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3302 00000100:00080000:3.0:1788204030.849207:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204030.849209:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204030.849213:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204030.849220:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064443031936/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204030.849245:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204030.849288:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204030/real 1788204030] req@ffff902352688380 x1875064443031552/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204030.849303:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443031552/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204030.849311:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204030.849313:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204030.849315:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443031552:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204030.849336:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443031680:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204030.849351:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204030.849358:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204030/real 1788204030] req@ffff90235268ad80 x1875064443031680/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204030.849366:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443031680/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204030.849371:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204030.849373:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204030.849375:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443031680:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204030.849378:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443031808:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204030.849387:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204030.849392:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204030/real 1788204030] req@ffff90235268b480 x1875064443031808/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204030.849398:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443031808/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204030.849413:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204030.849416:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204030.849417:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443031808:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204030.849421:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443031936:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204030.849433:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204030.849440:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204030/real 1788204030] req@ffff902352688a80 x1875064443031936/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204030.849446:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064443031936/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204030.849455:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204030.849457:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204030.849459:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443031936:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204031.681077:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204032.705047:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204033.728999:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204034.753063:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204035.777059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204035.968973:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204035.968980:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204035.968983:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204035.968991:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3307 00000100:00080000:3.0:1788204035.968999:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204035.969005:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204035.969036:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443032064/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204035.969056:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204035.969059:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204035.969060:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204035.969062:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3307 00000100:00080000:3.0:1788204035.969070:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204035.969072:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204035.969082:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204035.969092:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064443032192/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204035.969097:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443032064:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204035.969101:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204035.969105:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204035.969106:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204035.969109:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3307 00000100:00080000:3.0:1788204035.969111:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204035.969114:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204035.969117:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204035.969123:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443032320/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204035.969129:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204035.969131:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204035.969132:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204035.969134:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3307 00000100:00080000:3.0:1788204035.969142:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204035.969144:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204035.969146:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204035.969152:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443032448/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204035.969183:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443032192:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204035.969202:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443032320:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204035.969218:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443032448:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204036.801162:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204037.825973:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204038.849008:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788204039.041124:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:0.0:1788204039.041168:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788204039.041184:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788204039.041223:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788204039.041232:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788204039.041239:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:2.0:1788204039.041303:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204035/real 1788204039] req@ffff902352688a80 x1875064443032192/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204039.041332:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064443032192/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204039.041351:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204039.041356:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204039.041360:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443032192:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204039.041368:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204035/real 1788204039] req@ffff90235268b480 x1875064443032320/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204039.041380:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443032320/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204039.041387:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204039.041391:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204039.041395:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443032320:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204039.041402:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204035/real 1788204039] req@ffff90235268ad80 x1875064443032448/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204039.041413:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443032448/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204039.041421:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204039.041424:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204039.041427:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443032448:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204039.041434:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204035/real 1788204039] req@ffff90235268aa00 x1875064443032064/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204039.041446:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443032064/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204039.041453:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204039.041457:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204039.041460:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443032064:192.168.201.130@tcp:250:kworker.0 00010000:00010000:0.0:1788204039.872978:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204040.896990:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204041.088978:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204041.088986:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204041.088989:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204041.088998:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3312 00000100:00080000:3.0:1788204041.089005:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204041.089011:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204041.089036:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443032576/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204041.089058:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204041.089060:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204041.089061:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204041.089064:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3312 00000100:00080000:3.0:1788204041.089066:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204041.089068:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204041.089076:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204041.089087:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443032704/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204041.089096:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204041.089100:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204041.089102:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204041.089106:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3312 00000100:00080000:3.0:1788204041.089109:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204041.089113:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204041.089118:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443032576:192.168.201.130@tcp:250:kworker.0 00000008:00100000:3.0:1788204041.089120:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204041.089136:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443032832/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204041.089146:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204041.089150:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204041.089152:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204041.089155:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3312 00000100:00080000:3.0:1788204041.089173:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204041.089178:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:2.0:1788204041.089183:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:3.0:1788204041.089185:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204041.089207:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443032960/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204041.089240:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204041/real 1788204041] req@ffff902352688380 x1875064443032576/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204041.089260:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443032576/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204041.089271:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204041.089275:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204041.089281:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443032576:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204041.089288:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443032704:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204041.089310:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204041.089322:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204041/real 1788204041] req@ffff902352689880 x1875064443032704/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204041.089344:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443032704/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204041.089352:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204041.089356:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204041.089359:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443032704:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204041.089392:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443032832:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204041.089411:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204041.089421:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204041/real 1788204041] req@ffff90235268bb80 x1875064443032832/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204041.089431:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443032832/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204041.089439:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204041.089442:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204041.089445:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443032832:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204041.089463:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443032960:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204041.089479:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204041.089489:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204041/real 1788204041] req@ffff902352688700 x1875064443032960/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204041.089499:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443032960/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204041.089506:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204041.089509:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204041.089512:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443032960:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204041.920950:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204042.945044:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204043.969078:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204044.993022:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204046.017073:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204046.208992:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204046.208999:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204046.209002:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204046.209009:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3317 00000100:00080000:0.0:1788204046.209018:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204046.209023:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204046.209052:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443033088/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204046.209075:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204046.209078:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204046.209079:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204046.209081:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3317 00000100:00080000:0.0:1788204046.209083:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204046.209085:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204046.209091:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204046.209099:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443033216/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204046.209105:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204046.209107:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204046.209111:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204046.209114:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3317 00000100:00080000:0.0:1788204046.209115:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204046.209117:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204046.209120:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204046.209126:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443033344/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204046.209131:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204046.209134:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204046.209135:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204046.209153:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3317 00000100:00080000:0.0:1788204046.209156:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204046.209158:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204046.209162:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204046.209164:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443033088:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204046.209171:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443033472/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204046.209216:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204046.209245:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204046/real 1788204046] req@ffff9023450bad80 x1875064443033088/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204046.209258:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443033088/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204046.209265:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204046.209267:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204046.209270:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443033088:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204046.209275:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443033216:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204046.209287:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204046.209294:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204046/real 1788204046] req@ffff9023450b8700 x1875064443033216/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204046.209302:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443033216/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204046.209307:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204046.209310:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204046.209312:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443033216:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204046.209318:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443033344:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204046.209336:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204046.209347:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204046/real 1788204046] req@ffff9023450b9f80 x1875064443033344/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204046.209358:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443033344/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204046.209366:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204046.209369:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204046.209372:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443033344:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204046.209396:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443033472:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204046.209415:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204046.209423:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204046/real 1788204046] req@ffff9023450b9c00 x1875064443033472/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204046.209430:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443033472/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204046.209439:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204046.209441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204046.209443:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443033472:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204047.041030:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204048.065153:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204049.089026:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204050.113030:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204051.137018:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204051.328988:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204051.328996:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204051.328999:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204051.329006:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3322 00000100:00080000:1.0:1788204051.329014:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204051.329019:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204051.329069:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443033600/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204051.329090:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204051.329093:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204051.329094:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204051.329097:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3322 00000100:00080000:1.0:1788204051.329099:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204051.329101:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204051.329108:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204051.329117:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443033728/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204051.329125:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204051.329128:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204051.329130:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788204051.329130:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443033600:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204051.329138:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3322 00000100:00080000:1.0:1788204051.329140:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204051.329143:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204051.329145:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204051.329160:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443033856/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204051.329166:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204051.329169:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204051.329171:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204051.329174:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3322 00000100:00080000:1.0:1788204051.329179:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204051.329182:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204051.329187:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788204051.329192:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204051.329200:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443033984/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204051.329224:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204051/real 1788204051] req@ffff90234519ea00 x1875064443033600/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204051.329239:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443033600/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204051.329246:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204051.329248:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204051.329251:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443033600:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204051.329272:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443033728:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204051.329286:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204051.329294:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204051/real 1788204051] req@ffff90234519f100 x1875064443033728/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204051.329301:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443033728/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204051.329305:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204051.329312:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204051.329314:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443033728:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204051.329319:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443033856:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204051.329329:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204051.329335:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204051/real 1788204051] req@ffff90234519c380 x1875064443033856/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204051.329341:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443033856/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204051.329346:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204051.329348:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204051.329350:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443033856:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204051.329352:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443033984:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204051.329361:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204051.329366:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204051/real 1788204051] req@ffff90234519f480 x1875064443033984/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204051.329372:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443033984/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204051.329376:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204051.329377:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204051.329379:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443033984:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204052.161007:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204053.184999:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204054.209045:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204055.233060:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204056.257074:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204056.449083:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204056.449093:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204056.449098:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204056.449109:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3327 00000100:00080000:1.0:1788204056.449122:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204056.449129:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204056.449167:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064443034112/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204056.449190:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204056.449192:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204056.449194:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204056.449197:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3327 00000100:00080000:1.0:1788204056.449199:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204056.449203:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204056.449212:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204056.449223:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443034240/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204056.449232:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204056.449234:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443034112:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204056.449238:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204056.449242:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204056.449246:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3327 00000100:00080000:1.0:1788204056.449248:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204056.449251:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204056.449255:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204056.449265:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443034368/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204056.449272:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204056.449275:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204056.449276:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204056.449279:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3327 00000100:00080000:1.0:1788204056.449281:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204056.449283:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204056.449286:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204056.449293:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443034496/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204056.449317:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204056.449550:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204056/real 1788204056] req@ffff90234519dc00 x1875064443034112/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204056.449610:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064443034112/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204056.449622:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204056.449627:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204056.449631:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443034112:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204056.449657:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443034240:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204056.449683:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204056.449696:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204056/real 1788204056] req@ffff90234519f480 x1875064443034240/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204056.449710:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443034240/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204056.449718:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204056.449721:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204056.449726:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443034240:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204056.449733:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443034368:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204056.449753:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204056.449765:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204056/real 1788204056] req@ffff90234519c380 x1875064443034368/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204056.449778:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443034368/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204056.449786:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204056.449789:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204056.449793:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443034368:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204056.449799:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443034496:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204056.449818:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204056.449829:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204056/real 1788204056] req@ffff90234519f100 x1875064443034496/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204056.449841:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443034496/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204056.449848:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204056.449851:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204056.449855:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443034496:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204057.281098:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204058.305025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204059.329049:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204060.352998:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204061.377113:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204061.569040:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204061.569051:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204061.569055:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204061.569066:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3333 00000100:00080000:1.0:1788204061.569078:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204061.569085:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204061.569128:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443034624/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204061.569155:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204061.569158:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204061.569159:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204061.569162:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3333 00000100:00080000:1.0:1788204061.569164:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204061.569166:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204061.569174:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204061.569183:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443034752/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204061.569190:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204061.569192:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204061.569193:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204061.569195:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3333 00000100:00080000:1.0:1788204061.569197:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204061.569199:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204061.569202:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204061.569206:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443034624:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204061.569211:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443034880/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204061.569218:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204061.569221:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204061.569222:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204061.569224:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3333 00000100:00080000:1.0:1788204061.569226:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204061.569229:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204061.569231:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204061.569238:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443035008/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204061.569285:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204061.569334:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204061/real 1788204061] req@ffff90234519ea00 x1875064443034624/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204061.569351:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443034624/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204061.569359:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204061.569362:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204061.569365:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443034624:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204061.569371:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443034752:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204061.569387:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204061.569395:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204061/real 1788204061] req@ffff90234519c000 x1875064443034752/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204061.569402:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443034752/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204061.569407:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204061.569409:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204061.569411:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443034752:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204061.569431:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443034880:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204061.569442:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204061.569448:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204061/real 1788204061] req@ffff90234519c700 x1875064443034880/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204061.569455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443034880/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204061.569464:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204061.569466:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204061.569468:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443034880:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204061.569471:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443035008:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204061.569479:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204061.569484:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204061/real 1788204061] req@ffff90234519fb80 x1875064443035008/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204061.569490:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443035008/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204061.569495:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204061.569497:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204061.569500:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443035008:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204062.401019:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204063.425003:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204064.449022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204065.473046:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204066.497031:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204066.688945:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204066.688954:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204066.688959:0:8:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204066.688968:0:8:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3338 00000100:00080000:1.0:1788204066.688977:0:8:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204066.688983:0:8:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204066.689016:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443035136/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204066.689043:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204066.689047:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204066.689049:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204066.689053:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3338 00000100:00080000:1.0:1788204066.689055:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204066.689059:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204066.689067:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204066.689080:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443035136:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204066.689083:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443035264/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204066.689097:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204066.689100:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204066.689102:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204066.689112:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3338 00000100:00080000:1.0:1788204066.689115:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204066.689119:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204066.689123:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204066.689132:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443035392/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204066.689138:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204066.689141:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204066.689142:0:8:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204066.689144:0:8:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3338 00000100:00080000:1.0:1788204066.689147:0:8:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788204066.689151:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788204066.689152:0:8:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204066.689158:0:8:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204066.689164:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443035520/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204066.689192:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204066/real 1788204066] req@ffff90234519df80 x1875064443035136/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204066.689204:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443035136/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204066.689211:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204066.689214:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204066.689217:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443035136:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204066.689234:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443035264:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204066.689249:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204066.689257:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204066/real 1788204066] req@ffff90234519ed80 x1875064443035264/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204066.689264:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443035264/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204066.689269:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204066.689271:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204066.689273:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443035264:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204066.689276:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443035392:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204066.689289:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204066.689298:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204066/real 1788204066] req@ffff90234519d880 x1875064443035392/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204066.689309:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443035392/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204066.689317:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204066.689320:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204066.689323:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443035392:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204066.689329:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443035520:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204066.689345:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204066.689356:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204066/real 1788204066] req@ffff90234519e300 x1875064443035520/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204066.689367:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443035520/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204066.689374:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204066.689377:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204066.689380:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443035520:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204067.521007:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204068.545054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204069.569034:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204070.593026:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204071.617027:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204071.808986:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204071.808993:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204071.808996:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204071.809005:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3343 00000100:00080000:2.0:1788204071.809015:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204071.809030:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204071.809065:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443035648/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204071.809090:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204071.809094:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204071.809095:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204071.809099:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3343 00000100:00080000:2.0:1788204071.809101:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204071.809104:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204071.809111:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204071.809121:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443035776/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204071.809128:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204071.809131:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204071.809133:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788204071.809133:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443035648:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204071.809139:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3343 00000100:00080000:2.0:1788204071.809143:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204071.809146:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204071.809148:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204071.809156:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443035904/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204071.809166:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204071.809168:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204071.809169:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204071.809171:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3343 00000100:00080000:2.0:1788204071.809188:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204071.809190:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204071.809192:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788204071.809198:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204071.809200:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443036032/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204071.809235:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204071/real 1788204071] req@ffff90234598e300 x1875064443035648/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204071.809251:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443035648/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204071.809261:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204071.809265:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204071.809268:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443035648:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204071.809290:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443035776:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204071.809309:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204071.809319:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204071/real 1788204071] req@ffff90234598d180 x1875064443035776/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204071.809329:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443035776/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204071.809356:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204071.809359:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204071.809362:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443035776:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204071.809368:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443035904:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204071.809385:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204071.809395:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204071/real 1788204071] req@ffff90234598c380 x1875064443035904/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204071.809406:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443035904/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204071.809413:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204071.809417:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204071.809420:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443035904:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204071.809425:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443036032:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204071.809440:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204071.809448:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204071/real 1788204071] req@ffff90234598f480 x1875064443036032/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204071.809459:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443036032/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204071.809466:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204071.809468:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204071.809471:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443036032:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204072.640993:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204073.665024:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204074.689018:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204075.713004:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204076.737034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204076.928999:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204076.929009:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204076.929012:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204076.929020:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3348 00000100:00080000:0.0:1788204076.929029:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204076.929034:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204076.929064:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443036160/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204076.929085:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204076.929087:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204076.929088:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204076.929091:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3348 00000100:00080000:0.0:1788204076.929094:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204076.929097:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204076.929107:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204076.929119:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443036288/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204076.929124:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443036160:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204076.929136:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204076.929140:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204076.929142:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204076.929147:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3348 00000100:00080000:0.0:1788204076.929151:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204076.929154:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204076.929158:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204076.929167:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443036416/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204076.929173:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204076.929178:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1788204076.929178:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788204076.929181:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204076.929185:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3348 00000100:00080000:0.0:1788204076.929187:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204076.929190:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204076.929193:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204076.929202:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443036544/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204076.929212:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204076/real 1788204076] req@ffff9023450bb800 x1875064443036160/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204076.929226:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443036160/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204076.929233:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204076.929236:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204076.929239:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443036160:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204076.929255:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443036288:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204076.929268:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204076.929275:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204076/real 1788204076] req@ffff9023450b8a80 x1875064443036288/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204076.929282:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443036288/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204076.929286:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204076.929288:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204076.929290:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443036288:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204076.929293:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443036416:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204076.929302:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204076.929308:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204076/real 1788204076] req@ffff9023450b8380 x1875064443036416/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204076.929314:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443036416/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204076.929319:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204076.929321:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204076.929322:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443036416:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204076.929325:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443036544:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204076.929333:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204076.929338:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204076/real 1788204076] req@ffff9023450b9500 x1875064443036544/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204076.929344:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443036544/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204076.929348:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204076.929350:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204076.929352:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443036544:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204077.761015:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204078.785028:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204079.808998:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204080.833046:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204081.857002:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204082.048957:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204082.048964:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204082.048967:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204082.048975:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3353 00000100:00080000:0.0:1788204082.048983:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204082.048988:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204082.049023:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443036672/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204082.049041:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204082.049044:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204082.049045:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204082.049047:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3353 00000100:00080000:0.0:1788204082.049049:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204082.049052:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204082.049060:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204082.049075:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064443036800/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204082.049084:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204082.049087:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204082.049090:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204082.049094:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3353 00000100:00080000:0.0:1788204082.049098:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204082.049101:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443036672:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204082.049113:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204082.049120:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204082.049135:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064443036928/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204082.049145:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204082.049149:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204082.049151:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204082.049155:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3353 00000800:00000100:3.0:1788204082.049159:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788204082.049160:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204082.049167:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204082.049173:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204082.049185:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064443037056/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204082.049197:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204082/real 1788204082] req@ffff9023450bb480 x1875064443036672/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204082.049216:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443036672/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204082.049226:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204082.049237:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204082.049240:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443036672:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204082.049247:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443036800:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204082.049264:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204082.049272:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204082/real 1788204082] req@ffff9023450ba680 x1875064443036800/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204082.049279:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064443036800/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204082.049284:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204082.049286:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204082.049288:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443036800:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204082.049308:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443036928:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204082.049327:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204082.049336:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204082/real 1788204082] req@ffff9023450b8e00 x1875064443036928/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204082.049347:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064443036928/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204082.049354:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204082.049357:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204082.049360:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443036928:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204082.049366:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443037056:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204082.049382:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204082.049392:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204082/real 1788204082] req@ffff9023450bbb80 x1875064443037056/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204082.049403:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064443037056/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204082.049410:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204082.049413:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204082.049416:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443037056:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204082.881022:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204083.905017:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204084.929010:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204085.953025:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204086.977026:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204087.168988:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204087.168995:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204087.168998:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204087.169006:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3358 00000100:00080000:2.0:1788204087.169015:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204087.169019:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204087.169049:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443037184/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204087.169069:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204087.169071:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204087.169072:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204087.169075:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3358 00000100:00080000:2.0:1788204087.169077:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204087.169078:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204087.169086:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204087.169094:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443037312/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204087.169103:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204087.169107:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204087.169109:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204087.169112:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3358 00000100:00080000:2.0:1788204087.169116:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204087.169119:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443037184:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204087.169122:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204087.169131:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204087.169143:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443037440/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204087.169153:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204087.169156:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204087.169158:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204087.169162:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3358 00000100:00080000:2.0:1788204087.169172:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204087.169175:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204087.169179:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788204087.169180:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204087.169190:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443037568/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204087.169221:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204087/real 1788204087] req@ffff90234598ce00 x1875064443037184/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204087.169233:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443037184/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204087.169240:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204087.169242:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204087.169245:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443037184:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204087.169249:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443037312:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204087.169263:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204087.169270:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204087/real 1788204087] req@ffff90234598c700 x1875064443037312/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204087.169277:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443037312/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204087.169282:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204087.169284:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204087.169285:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443037312:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204087.169302:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443037440:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204087.169313:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204087.169318:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204087/real 1788204087] req@ffff90234598dc00 x1875064443037440/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204087.169325:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443037440/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204087.169330:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204087.169332:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204087.169334:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443037440:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204087.169337:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443037568:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204087.169344:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204087.169349:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204087/real 1788204087] req@ffff90234598f800 x1875064443037568/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204087.169355:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443037568/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204087.169363:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204087.169365:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204087.169367:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443037568:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204088.001029:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204089.025048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204090.048998:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204091.073021:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204092.097054:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204092.289008:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204092.289019:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204092.289024:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204092.289035:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3363 00000100:00080000:2.0:1788204092.289046:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204092.289054:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204092.289102:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064443037696/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204092.289128:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204092.289131:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204092.289132:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204092.289135:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3363 00000100:00080000:2.0:1788204092.289137:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204092.289140:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204092.289149:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204092.289160:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064443037824/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204092.289167:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788204092.289169:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443037696:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204092.289170:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204092.289173:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204092.289179:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3363 00000100:00080000:2.0:1788204092.289182:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204092.289184:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204092.289189:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204092.289201:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064443037952/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204092.289217:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204092.289221:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204092.289224:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204092.289228:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3363 00000100:00080000:2.0:1788204092.289232:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204092.289237:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204092.289241:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204092.289254:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204092.289255:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443038080/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204092.289305:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204092/real 1788204092] req@ffff90234598ca80 x1875064443037696/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204092.289328:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064443037696/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204092.289356:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204092.289361:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204092.289365:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443037696:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204092.289394:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443037824:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204092.289419:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204092.289432:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204092/real 1788204092] req@ffff90234598e680 x1875064443037824/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204092.289445:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064443037824/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204092.289453:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204092.289457:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204092.289460:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443037824:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204092.289466:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443037952:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204092.289485:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204092.289495:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204092/real 1788204092] req@ffff90234598d880 x1875064443037952/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204092.289506:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064443037952/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204092.289521:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204092.289525:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204092.289529:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443037952:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204092.289535:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443038080:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204092.289552:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204092.289565:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204092/real 1788204092] req@ffff90234598f800 x1875064443038080/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204092.289589:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443038080/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204092.289605:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204092.289610:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204092.289615:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443038080:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204093.121060:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204094.145039:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204095.169042:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204096.193073:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204097.217066:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204097.409003:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204097.409014:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204097.409019:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204097.409030:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3368 00000100:00080000:0.0:1788204097.409042:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204097.409049:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204097.409099:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064443038208/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204097.409132:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204097.409137:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204097.409139:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204097.409144:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3368 00000100:00080000:0.0:1788204097.409148:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204097.409153:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204097.409163:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204097.409170:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443038208:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204097.409179:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064443038336/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204097.409193:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204097.409197:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204097.409199:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204097.409204:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3368 00000100:00080000:0.0:1788204097.409209:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204097.409213:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204097.409219:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204097.409231:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064443038464/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204097.409234:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788204097.409247:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204097.409266:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204097.409268:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204097.409274:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3368 00000100:00080000:0.0:1788204097.409279:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204097.409283:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204097.409289:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204097.409297:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204097/real 1788204097] req@ffff9023450bbb80 x1875064443038208/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204097.409304:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443038592/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204097.409335:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064443038208/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204097.409359:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204097.409364:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204097.409368:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443038208:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204097.409398:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443038336:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204097.409421:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204097.409435:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204097/real 1788204097] req@ffff9023450b8e00 x1875064443038336/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204097.409448:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064443038336/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204097.409457:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204097.409460:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204097.409463:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443038336:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204097.409470:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443038464:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204097.409489:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204097.409500:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204097/real 1788204097] req@ffff9023450ba680 x1875064443038464/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204097.409512:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064443038464/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204097.409519:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204097.409522:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204097.409525:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443038464:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204097.409530:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443038592:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204097.409548:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204097.409559:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204097/real 1788204097] req@ffff9023450bb480 x1875064443038592/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204097.409569:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443038592/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204097.409576:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204097.409580:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204097.409583:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443038592:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204098.241038:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204099.265025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204100.289051:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204101.313000:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204102.337059:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204102.528974:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204102.528981:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204102.528985:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204102.528993:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3374 00000100:00080000:0.0:1788204102.529003:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204102.529008:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204102.529041:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443038720/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204102.529066:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204102.529069:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204102.529070:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204102.529073:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3374 00000100:00080000:0.0:1788204102.529076:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204102.529080:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204102.529089:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204102.529103:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443038848/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204102.529114:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204102.529118:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204102.529120:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204102.529125:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3374 00000100:00080000:0.0:1788204102.529129:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204102.529131:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443038720:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204102.529136:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204102.529143:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204102.529154:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443038976/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204102.529176:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204102.529179:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204102.529180:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204102.529184:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3374 00000100:00080000:0.0:1788204102.529187:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204102.529190:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204102.529195:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204102.529214:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443039104/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204102.529245:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443038848:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204102.529289:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443038976:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204102.529313:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443039104:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204103.361096:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204104.385025:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204105.409052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788204105.601205:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788204105.601262:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788204105.601284:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788204105.601340:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788204105.601351:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788204105.601359:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788204105.601455:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204102/real 1788204105] req@ffff9023450b8a80 x1875064443038976/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204105.601484:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443038976/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204105.601500:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204105.601506:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204105.601510:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443038976:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204105.601519:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204102/real 1788204105] req@ffff9023450bb800 x1875064443039104/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204105.601534:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443039104/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204105.601541:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204105.601545:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204105.601548:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443039104:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204105.601555:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204102/real 1788204105] req@ffff9023450b9500 x1875064443038720/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204105.601567:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443038720/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204105.601580:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204105.601584:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204105.601587:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443038720:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204105.601594:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204102/real 1788204105] req@ffff9023450b8380 x1875064443038848/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204105.601605:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443038848/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204105.601612:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204105.601615:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204105.601618:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443038848:192.168.201.130@tcp:38:kworker.0 00010000:00010000:0.0:1788204106.433017:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204107.457002:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204107.649028:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204107.649036:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204107.649039:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204107.649049:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3379 00000100:00080000:1.0:1788204107.649059:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204107.649067:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204107.649113:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064443039232/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204107.649147:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204107.649151:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204107.649154:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204107.649158:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3379 00000100:00080000:1.0:1788204107.649163:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204107.649167:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204107.649176:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204107.649192:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443039360/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204107.649204:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204107.649209:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788204107.649211:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443039232:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204107.649212:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204107.649223:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3379 00000100:00080000:1.0:1788204107.649227:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204107.649231:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204107.649242:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204107.649255:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443039488/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204107.649265:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204107.649268:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204107.649271:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204107.649274:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3379 00000100:00080000:1.0:1788204107.649278:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204107.649285:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:3.0:1788204107.649286:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:1.0:1788204107.649292:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204107.649306:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443039616/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204107.649334:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204107/real 1788204107] req@ffff90234519f800 x1875064443039232/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204107.649357:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064443039232/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204107.649369:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204107.649373:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204107.649377:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443039232:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204107.649402:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443039360:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204107.649428:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204107.649442:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204107/real 1788204107] req@ffff90234519e300 x1875064443039360/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204107.649456:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443039360/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204107.649465:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204107.649469:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204107.649472:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443039360:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204107.649479:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443039488:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204107.649499:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204107.650378:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204107/real 1788204107] req@ffff90234519d880 x1875064443039488/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204107.650400:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443039488/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204107.650411:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204107.650414:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204107.650419:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443039488:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204107.650428:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443039616:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204107.650463:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204107.650479:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204107/real 1788204107] req@ffff90234519ed80 x1875064443039616/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204107.650500:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443039616/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204107.650509:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204107.650520:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204107.650524:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443039616:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204108.481011:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204109.505094:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204110.529093:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204111.553059:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204112.577128:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204112.769009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204112.769020:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204112.769025:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204112.769035:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3384 00000100:00080000:1.0:1788204112.769047:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204112.769055:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204112.769103:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443039744/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204112.769126:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204112.769129:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204112.769130:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204112.769133:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3384 00000100:00080000:1.0:1788204112.769134:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204112.769137:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204112.769148:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204112.769161:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443039872/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204112.769172:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204112.769175:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443039744:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204112.769176:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204112.769179:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204112.769188:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3384 00000100:00080000:1.0:1788204112.769191:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204112.769195:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204112.769201:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204112.769212:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443040000/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204112.769222:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204112.769226:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204112.769228:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204112.769232:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3384 00000100:00080000:1.0:1788204112.769236:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204112.769240:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204112.769245:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204112.769256:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443040128/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204112.769260:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204112.769310:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204112/real 1788204112] req@ffff90234519df80 x1875064443039744/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204112.769327:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443039744/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204112.769347:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204112.769351:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204112.769355:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443039744:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204112.769380:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443039872:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204112.769401:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204112.769412:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204112/real 1788204112] req@ffff90234519fb80 x1875064443039872/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204112.769422:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443039872/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204112.769434:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204112.769437:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204112.769440:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443039872:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204112.769446:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443040000:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204112.769461:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204112.769471:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204112/real 1788204112] req@ffff90234519c700 x1875064443040000/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204112.769481:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443040000/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204112.769488:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204112.769491:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204112.769493:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443040000:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204112.769498:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443040128:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204112.769514:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204112.769523:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204112/real 1788204112] req@ffff90234519c000 x1875064443040128/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204112.769532:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443040128/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204112.769540:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204112.769542:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204112.769546:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443040128:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204113.601146:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204114.625067:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204115.649032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204116.673082:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204117.697115:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204117.889045:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204117.889054:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204117.889058:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204117.889069:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3389 00000100:00080000:2.0:1788204117.889078:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204117.889084:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204117.889118:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443040256/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204117.889148:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204117.889150:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204117.889152:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204117.889161:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3389 00000100:00080000:2.0:1788204117.889164:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204117.889167:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204117.889175:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204117.889184:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443040384/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204117.889193:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204117.889195:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204117.889197:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204117.889203:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3389 00000100:00080000:2.0:1788204117.889208:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204117.889208:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443040256:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204117.889221:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204117.889230:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204117.889247:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443040512/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204117.889258:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204117.889261:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204117.889263:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204117.889267:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3389 00000100:00080000:2.0:1788204117.889285:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204117.889289:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204117.889294:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788204117.889301:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204117.889312:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443040640/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204117.889353:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204117/real 1788204117] req@ffff90234598dc00 x1875064443040256/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204117.889370:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443040256/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204117.889379:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204117.889382:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204117.889385:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443040256:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204117.889411:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443040384:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204117.889428:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204117.889438:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204117/real 1788204117] req@ffff90234598c700 x1875064443040384/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204117.889445:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443040384/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204117.889451:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204117.889453:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204117.889455:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443040384:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204117.889459:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443040512:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204117.889471:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204117.889478:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204117/real 1788204117] req@ffff90234598ce00 x1875064443040512/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204117.889486:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443040512/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204117.889492:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204117.889494:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204117.889496:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443040512:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204117.889499:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443040640:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204117.889520:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204117.889528:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204117/real 1788204117] req@ffff90234598f480 x1875064443040640/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204117.889536:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443040640/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204117.889542:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204117.889544:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204117.889546:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443040640:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204118.721053:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204119.745109:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204120.769031:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204121.793227:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204122.817038:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204123.008954:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204123.008962:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204123.008966:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204123.008974:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3394 00000100:00080000:2.0:1788204123.008983:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204123.008988:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204123.009024:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443040768/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204123.009066:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204123.009070:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204123.009073:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204123.009077:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3394 00000100:00080000:2.0:1788204123.009082:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204123.009087:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204123.009097:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204123.009110:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443040768:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204123.009111:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443040896/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204123.009120:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204123.009139:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204123.009142:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204123.009147:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3394 00000100:00080000:2.0:1788204123.009152:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204123.009157:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204123.009162:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204123.009176:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443041024/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204123.009185:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788204123.009185:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788204123.009191:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204123.009192:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204123.009197:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3394 00000100:00080000:2.0:1788204123.009201:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204123.009204:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204123.009208:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204123.009217:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443041152/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204123.009226:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204123/real 1788204123] req@ffff90234598c380 x1875064443040768/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204123.009249:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443040768/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204123.009257:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204123.009261:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204123.009264:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443040768:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204123.009283:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443040896:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204123.009300:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204123.009315:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204123/real 1788204123] req@ffff90234598d180 x1875064443040896/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204123.009324:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443040896/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204123.009329:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204123.009331:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204123.009334:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443040896:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204123.009337:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443041024:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204123.009353:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204123.009364:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204123/real 1788204123] req@ffff90234598e300 x1875064443041024/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204123.009376:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443041024/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204123.009385:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204123.009389:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204123.009393:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443041024:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204123.009400:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443041152:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204123.009418:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204123.009427:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204123/real 1788204123] req@ffff90234598df80 x1875064443041152/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204123.009435:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443041152/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204123.009440:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204123.009442:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204123.009444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443041152:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204123.841082:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204124.865086:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204125.889079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204126.913054:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204127.937107:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204128.129002:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204128.129026:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204128.129037:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204128.129050:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3399 00000100:00080000:2.0:1788204128.129065:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204128.129074:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204128.129123:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064443041280/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204128.129159:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204128.129164:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204128.129167:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204128.129172:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3399 00000100:00080000:2.0:1788204128.129177:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204128.129182:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204128.129195:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443041280:192.168.201.130@tcp:250:kworker.0 00000008:00100000:2.0:1788204128.129195:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204128.129218:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064443041408/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204128.129227:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204128.129231:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204128.129233:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204128.129237:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3399 00000100:00080000:2.0:1788204128.129244:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204128.129250:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204128.129257:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204128.129274:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204128.129275:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064443041536/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204128.129290:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204128.129295:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204128.129297:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204128.129301:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3399 00000100:00080000:2.0:1788204128.129307:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204128.129313:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204128.129318:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204128/real 1788204128] req@ffff90234598fb80 x1875064443041280/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788204128.129322:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204128.129341:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064443041280/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204128.129344:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443041664/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204128.129359:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204128.129367:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204128.129373:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443041280:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204128.129405:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443041408:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204128.129430:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204128.129442:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204128/real 1788204128] req@ffff90234598ed80 x1875064443041408/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204128.129456:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064443041408/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204128.129463:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204128.129465:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204128.129468:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443041408:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204128.129473:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443041536:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204128.129486:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204128.129493:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204128/real 1788204128] req@ffff90234598f100 x1875064443041536/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204128.129505:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064443041536/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204128.129514:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204128.129517:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204128.129521:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443041536:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204128.129527:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443041664:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204128.129550:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204128.129559:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204128/real 1788204128] req@ffff90234598df80 x1875064443041664/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204128.129568:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443041664/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204128.129578:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204128.129580:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204128.129583:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443041664:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204128.961057:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204129.985063:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204131.009009:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204132.033051:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204133.057070:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204133.249017:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204133.249026:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204133.249029:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204133.249038:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3404 00000100:00080000:1.0:1788204133.249047:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204133.249059:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204133.249094:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443041792/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204133.249117:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204133.249121:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204133.249122:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204133.249126:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3404 00000100:00080000:1.0:1788204133.249128:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204133.249131:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204133.249140:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204133.249150:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443041920/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204133.249160:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204133.249163:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204133.249164:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204133.249170:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443041792:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204133.249171:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3404 00000100:00080000:1.0:1788204133.249179:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204133.249183:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204133.249187:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204133.249197:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443042048/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204133.249205:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204133.249207:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204133.249209:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204133.249211:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3404 00000100:00080000:1.0:1788204133.249224:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204133.249228:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204133.249231:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204133.249239:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204133.249244:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443042176/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204133.249281:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204133/real 1788204133] req@ffff90234519ea00 x1875064443041792/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204133.249297:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443041792/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204133.249307:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204133.249310:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204133.249313:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443041792:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204133.249343:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443041920:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204133.249361:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204133.249371:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204133/real 1788204133] req@ffff90234519f100 x1875064443041920/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204133.249381:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443041920/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204133.249393:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204133.249396:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204133.249398:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443041920:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204133.249403:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443042048:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204133.249415:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204133.249422:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204133/real 1788204133] req@ffff90234519c380 x1875064443042048/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204133.249430:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443042048/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204133.249437:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204133.249439:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204133.249441:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443042048:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204133.249445:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443042176:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204133.249456:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204133.249463:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204133/real 1788204133] req@ffff90234519f480 x1875064443042176/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204133.249471:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443042176/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204133.249477:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204133.249479:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204133.249482:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443042176:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204134.081036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204135.105016:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204136.129028:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204137.153012:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204138.177016:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204138.369006:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204138.369017:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204138.369021:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204138.369030:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3409 00000100:00080000:1.0:1788204138.369039:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204138.369045:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204138.369077:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064443042304/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204138.369101:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204138.369103:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204138.369105:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204138.369108:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3409 00000100:00080000:1.0:1788204138.369110:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204138.369113:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204138.369120:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204138.369136:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443042432/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204138.369147:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443042304:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204138.369150:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204138.369156:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204138.369158:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204138.369163:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3409 00000100:00080000:1.0:1788204138.369168:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204138.369173:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204138.369179:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204138.369190:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443042560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204138.369202:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788204138.369207:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788204138.369208:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204138.369212:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204138.369217:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3409 00000100:00080000:1.0:1788204138.369220:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204138.369223:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204138.369226:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204138.369234:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443042688/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204138.369259:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204138/real 1788204138] req@ffff90234519dc00 x1875064443042304/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204138.369276:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064443042304/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204138.369284:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204138.369286:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204138.369290:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443042304:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204138.369308:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443042432:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204138.369326:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204138.369335:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204138/real 1788204138] req@ffff90234519f480 x1875064443042432/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204138.369343:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443042432/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204138.369348:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204138.369351:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204138.369353:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443042432:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204138.369357:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443042560:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204138.369368:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204138.369374:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204138/real 1788204138] req@ffff90234519c380 x1875064443042560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204138.369382:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443042560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204138.369387:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204138.369389:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204138.369391:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443042560:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204138.369394:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443042688:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204138.369403:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204138.369408:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204138/real 1788204138] req@ffff90234519f100 x1875064443042688/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204138.369416:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443042688/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204138.369421:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204138.369422:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204138.369424:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443042688:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204139.201051:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204140.225002:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204141.249116:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204142.273099:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204143.297107:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204143.489046:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204143.489062:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204143.489068:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204143.489079:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3415 00000100:00080000:3.0:1788204143.489092:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204143.489100:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204143.489153:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443042816/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204143.489186:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204143.489191:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204143.489193:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204143.489198:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3415 00000100:00080000:3.0:1788204143.489202:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204143.489209:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204143.489221:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204143.489237:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443042816:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204143.489238:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443042944/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204143.489255:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204143.489259:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204143.489261:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204143.489266:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3415 00000100:00080000:3.0:1788204143.489270:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204143.489275:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204143.489286:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204143.489298:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443043072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204143.489309:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204143.489313:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204143.489315:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204143.489320:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3415 00000800:00000100:0.0:1788204143.489324:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204143.489327:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204143.489335:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204143.489341:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204143.489355:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443043200/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204143.489371:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204143/real 1788204143] req@ffff902352689180 x1875064443042816/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204143.489389:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443042816/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204143.489398:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204143.489402:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204143.489405:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443042816:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204143.489432:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443042944:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204143.489473:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204143.489488:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204143/real 1788204143] req@ffff902352689c00 x1875064443042944/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204143.489501:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443042944/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204143.489508:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204143.489510:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204143.489513:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443042944:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204143.489518:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443043072:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204143.489534:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204143.489542:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204143/real 1788204143] req@ffff90235268a680 x1875064443043072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204143.489554:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443043072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204143.489560:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204143.489562:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204143.489564:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443043072:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204143.489567:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443043200:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204143.489578:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204143.489584:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204143/real 1788204143] req@ffff902352688e00 x1875064443043200/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204143.489591:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443043200/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204143.489597:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204143.489598:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204143.489600:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443043200:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204144.321083:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204145.345080:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204146.369321:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204147.393066:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204148.417055:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204148.608969:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204148.608985:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204148.608990:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204148.609003:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3420 00000100:00080000:3.0:1788204148.609017:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204148.609026:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204148.609077:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443043328/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204148.609123:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204148.609127:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204148.609129:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204148.609134:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3420 00000100:00080000:3.0:1788204148.609139:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204148.609144:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204148.609156:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204148.609172:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443043328:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204148.609173:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443043456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204148.609189:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204148.609193:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204148.609195:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204148.609200:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3420 00000100:00080000:3.0:1788204148.609203:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204148.609208:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204148.609213:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204148.609226:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443043584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204148.609237:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204148.609241:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204148.609243:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204148.609248:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3420 00000100:00080000:3.0:1788204148.609252:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788204148.609256:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204148.609261:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204148.609267:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204148.609281:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064443043712/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204148.609307:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204148/real 1788204148] req@ffff902352689500 x1875064443043328/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204148.609327:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443043328/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204148.609337:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204148.609341:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204148.609345:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443043328:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204148.609390:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443043456:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204148.609417:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204148.609428:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204148/real 1788204148] req@ffff90235268b800 x1875064443043456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204148.609437:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443043456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204148.609443:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204148.609445:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204148.609448:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443043456:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204148.609452:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443043584:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204148.609466:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204148.609472:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204148/real 1788204148] req@ffff90235268a300 x1875064443043584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204148.609480:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443043584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204148.609490:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204148.609491:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204148.609493:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443043584:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204148.609497:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443043712:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204148.609506:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204148.609512:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204148/real 1788204148] req@ffff90235268b100 x1875064443043712/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204148.609519:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064443043712/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204148.609524:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204148.609526:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204148.609528:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443043712:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204149.441092:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204150.465048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204151.489052:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204152.514652:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204153.537040:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204153.728920:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204153.728930:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204153.728933:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204153.728942:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3425 00000100:00080000:3.0:1788204153.728952:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204153.728958:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204153.729000:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443043840/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204153.729024:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204153.729027:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204153.729029:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204153.729032:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3425 00000100:00080000:3.0:1788204153.729036:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204153.729040:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204153.729049:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204153.729058:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443043840:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204153.729064:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064443043968/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204153.729076:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204153.729079:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204153.729080:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204153.729084:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3425 00000100:00080000:3.0:1788204153.729087:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204153.729091:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204153.729095:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204153.729104:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443044096/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204153.729112:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788204153.729116:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204153.729118:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204153.729121:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204153.729125:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3425 00000100:00080000:3.0:1788204153.729128:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204153.729131:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204153.729136:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204153.729153:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443044224/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204153.729158:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204153/real 1788204153] req@ffff902352688000 x1875064443043840/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204153.729179:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443043840/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204153.729196:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204153.729199:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204153.729202:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443043840:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204153.729219:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443043968:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204153.729235:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204153.729243:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204153/real 1788204153] req@ffff90235268b100 x1875064443043968/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204153.729250:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064443043968/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204153.729255:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204153.729256:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204153.729258:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443043968:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204153.729261:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443044096:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204153.729281:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204153.729290:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204153/real 1788204153] req@ffff90235268a300 x1875064443044096/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204153.729299:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443044096/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204153.729307:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204153.729310:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204153.729313:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443044096:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204153.729319:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443044224:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204153.729337:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204153.729347:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204153/real 1788204153] req@ffff90235268b800 x1875064443044224/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204153.729355:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443044224/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204153.729362:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204153.729365:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204153.729368:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443044224:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204154.561065:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204155.585046:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204156.609049:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204157.633041:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204158.657001:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204158.849008:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204158.849019:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204158.849022:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204158.849031:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3430 00000100:00080000:0.0:1788204158.849041:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204158.849048:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204158.849095:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443044352/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204158.849125:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204158.849128:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204158.849131:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204158.849136:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3430 00000100:00080000:0.0:1788204158.849139:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204158.849144:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204158.849151:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443044352:192.168.201.130@tcp:250:kworker.0 00000008:00100000:0.0:1788204158.849153:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204158.849177:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443044480/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204158.849202:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204158.849206:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204158.849208:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204158.849212:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3430 00000100:00080000:0.0:1788204158.849216:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788204158.849217:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788204158.849222:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204158.849228:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204158.849240:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443044608/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204158.849248:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204158.849257:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788204158.849261:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204158/real 1788204158] req@ffff9023450b9c00 x1875064443044352/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204158.849264:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204158.849272:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3430 00000100:00080000:0.0:1788204158.849276:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204158.849282:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204158.849286:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204158.849289:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443044352/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204158.849298:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788204158.849299:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443044736/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204158.849304:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204158.849311:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443044352:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204158.849332:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443044480:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204158.849359:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204158.849372:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204158/real 1788204158] req@ffff9023450b9f80 x1875064443044480/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204158.849384:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443044480/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204158.849392:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204158.849395:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204158.849398:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443044480:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204158.849404:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443044608:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204158.849423:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204158.849433:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204158/real 1788204158] req@ffff9023450b8700 x1875064443044608/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204158.849451:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443044608/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204158.849461:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204158.849463:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204158.849464:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443044608:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204158.849468:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443044736:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204158.849482:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204158.849494:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204158/real 1788204158] req@ffff9023450bad80 x1875064443044736/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204158.849501:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443044736/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204158.849505:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204158.849507:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204158.849509:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443044736:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204159.681024:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204160.705081:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204161.729088:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204162.753090:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204163.777068:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204163.969016:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204163.969029:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204163.969034:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204163.969047:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3435 00000100:00080000:0.0:1788204163.969061:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204163.969069:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204163.969128:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443044864/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204163.969162:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204163.969165:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204163.969167:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204163.969171:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3435 00000100:00080000:0.0:1788204163.969175:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204163.969180:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204163.969191:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204163.969212:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443044992/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204163.969212:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443044864:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204163.969228:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204163.969231:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204163.969233:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204163.969236:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3435 00000100:00080000:0.0:1788204163.969240:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204163.969243:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204163.969247:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204163.969261:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443045120/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204163.969271:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204163.969275:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204163.969277:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204163.969281:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3435 00000100:00080000:0.0:1788204163.969286:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204163.969291:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204163.969296:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788204163.969309:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204163.969310:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443045248/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204163.969372:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204163/real 1788204163] req@ffff9023450ba300 x1875064443044864/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204163.969472:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443044864/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204163.969484:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204163.969616:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204163.969624:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443044864:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204163.969667:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443044992:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204163.969697:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204163.969711:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204163/real 1788204163] req@ffff9023450b9180 x1875064443044992/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204163.969740:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443044992/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204163.969749:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204163.969752:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204163.969756:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443044992:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204163.969764:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443045120:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204163.969791:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204163.969808:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204163/real 1788204163] req@ffff9023450b8000 x1875064443045120/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204163.969822:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443045120/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204163.969839:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204163.969844:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204163.969848:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443045120:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204163.969855:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443045248:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204163.969951:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204163.969962:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204163/real 1788204163] req@ffff9023450baa00 x1875064443045248/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204163.969971:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443045248/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204163.969984:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204163.969987:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204163.969990:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443045248:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204164.801040:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204165.825104:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204166.849056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204167.872990:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204168.897091:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204169.089016:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204169.089029:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204169.089034:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204169.089047:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3440 00000100:00080000:0.0:1788204169.089061:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204169.089069:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204169.089119:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443045376/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204169.089156:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204169.089160:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204169.089162:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204169.089168:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3440 00000100:00080000:0.0:1788204169.089173:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204169.089179:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204169.089192:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204169.089200:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443045376:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204169.089211:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443045504/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204169.089224:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204169.089228:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204169.089230:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204169.089235:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3440 00000100:00080000:0.0:1788204169.089252:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204169.089257:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204169.089264:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204169.089280:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443045632/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204169.089291:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204169.089301:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204169.089304:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204169.089310:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3440 00000100:00080000:0.0:1788204169.089315:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204169.089321:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204169.089324:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443045504:192.168.201.130@tcp:38:kworker.0 00000008:00100000:0.0:1788204169.089329:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204169.089346:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443045760/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204169.089355:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443045632:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204169.089389:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443045760:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204169.921399:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204170.945598:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204171.969044:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788204172.161198:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:0.0:1788204172.161240:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788204172.161266:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788204172.161307:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788204172.161317:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788204172.161324:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788204172.161435:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204169/real 1788204172] req@ffff9023450ba300 x1875064443045760/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204172.161458:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443045760/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204172.161469:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204172.161474:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204172.161478:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443045760:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204172.161486:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204169/real 1788204172] req@ffff9023450b8000 x1875064443045504/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204172.161504:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443045504/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204172.161512:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204172.161515:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204172.161518:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443045504:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204172.161525:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204169/real 1788204172] req@ffff9023450b9180 x1875064443045632/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204172.161536:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443045632/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204172.161544:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204172.161547:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204172.161550:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443045632:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204172.161557:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204169/real 1788204172] req@ffff9023450baa00 x1875064443045376/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204172.161568:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443045376/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204172.161575:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204172.161579:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204172.161583:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443045376:192.168.201.130@tcp:250:kworker.0 00010000:00010000:2.0:1788204172.993431:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204174.017041:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204174.209110:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204174.209123:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204174.209128:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204174.209142:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3445 00000100:00080000:1.0:1788204174.209163:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204174.209171:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204174.209216:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443045888/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204174.209251:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204174.209255:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204174.209257:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204174.209262:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3445 00000100:00080000:1.0:1788204174.209265:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204174.209269:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204174.209286:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204174.209299:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443046016/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204174.209311:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204174.209315:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788204174.209315:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443045888:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204174.209318:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204174.209328:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3445 00000100:00080000:1.0:1788204174.209331:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204174.209335:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204174.209341:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204174.209353:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443046144/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204174.209363:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204174.209367:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204174.209369:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204174.209373:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3445 00000100:00080000:1.0:1788204174.209377:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204174.209382:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204174.209386:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204174.209398:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443046272/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204174.209409:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204174.209470:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204174/real 1788204174] req@ffff90234519ea00 x1875064443045888/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204174.209503:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443045888/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204174.209513:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204174.209517:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204174.209521:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443045888:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204174.209549:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443046016:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204174.209572:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204174.209586:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204174/real 1788204174] req@ffff90234519c000 x1875064443046016/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204174.209598:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443046016/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204174.209606:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204174.209613:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204174.209617:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443046016:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204174.209623:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443046144:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204174.209642:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204174.209653:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204174/real 1788204174] req@ffff90234519c700 x1875064443046144/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204174.209664:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443046144/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204174.209677:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204174.209680:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204174.209683:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443046144:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204174.209688:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443046272:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204174.209702:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204174.209711:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204174/real 1788204174] req@ffff90234519fb80 x1875064443046272/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204174.209721:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443046272/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204174.209727:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204174.209730:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204174.209733:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443046272:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204175.041059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204176.065082:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204177.089040:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204178.113083:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204179.137000:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204179.328982:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204179.328989:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204179.328992:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204179.329000:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3450 00000100:00080000:0.0:1788204179.329008:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204179.329014:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204179.329049:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443046400/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204179.329076:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204179.329080:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204179.329082:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204179.329085:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3450 00000100:00080000:0.0:1788204179.329093:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204179.329097:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204179.329106:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204179.329118:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443046528/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204179.329124:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443046400:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204179.329134:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204179.329138:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204179.329139:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204179.329143:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3450 00000100:00080000:0.0:1788204179.329146:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204179.329148:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204179.329152:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204179.329162:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443046656/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204179.329169:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204179.329172:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204179.329173:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204179.329175:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3450 00000100:00080000:0.0:1788204179.329178:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204179.329180:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204179.329182:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204179.329189:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443046784/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204179.329195:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204179.329228:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204179/real 1788204179] req@ffff9023450bad80 x1875064443046400/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204179.329240:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443046400/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204179.329247:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204179.329249:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204179.329252:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443046400:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204179.329268:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443046528:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204179.329282:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204179.329290:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204179/real 1788204179] req@ffff9023450b8700 x1875064443046528/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204179.329297:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443046528/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204179.329302:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204179.329303:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204179.329305:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443046528:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204179.329308:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443046656:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204179.329318:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204179.329323:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204179/real 1788204179] req@ffff9023450b9f80 x1875064443046656/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204179.329330:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443046656/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204179.329335:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204179.329336:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204179.329338:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443046656:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204179.329341:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443046784:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204179.329358:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204179.329364:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204179/real 1788204179] req@ffff9023450b9c00 x1875064443046784/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204179.329372:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443046784/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204179.329376:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204179.329378:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204179.329380:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443046784:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204180.160982:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204181.185047:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204182.209082:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204183.233066:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204184.257112:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204184.449003:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204184.449012:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204184.449016:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204184.449025:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3455 00000100:00080000:0.0:1788204184.449036:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204184.449044:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204184.449082:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443046912/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204184.449112:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204184.449115:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204184.449118:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204184.449122:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3455 00000100:00080000:0.0:1788204184.449125:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204184.449129:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204184.449139:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204184.449152:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443047040/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204184.449161:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204184.449164:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204184.449165:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204184.449169:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3455 00000100:00080000:0.0:1788204184.449173:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204184.449177:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204184.449181:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204184.449189:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443047168/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204184.449197:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204184.449200:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204184.449201:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204184.449204:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3455 00000100:00080000:0.0:1788204184.449206:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204184.449211:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204184.449216:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204184.449216:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443046912:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204184.449225:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443047296/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204184.449316:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204184.449371:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204184/real 1788204184] req@ffff9023450b8380 x1875064443046912/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204184.449393:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443046912/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204184.449405:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204184.449410:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204184.449414:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443046912:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204184.449424:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443047040:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204184.449449:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204184.449464:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204184/real 1788204184] req@ffff9023450b9500 x1875064443047040/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204184.449478:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443047040/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204184.449494:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204184.449498:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204184.449502:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443047040:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204184.449533:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443047168:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204184.449556:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204184.449568:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204184/real 1788204184] req@ffff9023450bb800 x1875064443047168/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204184.449581:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443047168/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204184.449590:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204184.449594:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204184.449597:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443047168:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204184.449604:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443047296:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204184.449637:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204184.449650:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204184/real 1788204184] req@ffff9023450b8a80 x1875064443047296/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204184.449662:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443047296/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204184.449671:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204184.449674:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204184.449678:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443047296:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204185.281014:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204186.305024:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204187.329019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204188.353033:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204189.377037:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204189.569004:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204189.569011:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204189.569014:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204189.569021:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3461 00000100:00080000:1.0:1788204189.569030:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204189.569034:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204189.569065:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443047424/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204189.569089:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204189.569093:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204189.569094:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204189.569099:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3461 00000100:00080000:1.0:1788204189.569101:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204189.569105:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204189.569114:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204189.569126:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443047552/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204189.569134:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204189.569138:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204189.569139:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788204189.569139:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443047424:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204189.569142:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3461 00000100:00080000:1.0:1788204189.569145:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204189.569148:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204189.569152:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204189.569159:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443047680/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204189.569166:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204189.569168:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204189.569170:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204189.569172:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3461 00000100:00080000:1.0:1788204189.569175:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204189.569178:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204189.569182:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204189.569189:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443047808/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204189.569195:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204189.569232:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204189/real 1788204189] req@ffff90234519df80 x1875064443047424/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204189.569249:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443047424/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204189.569261:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204189.569264:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204189.569266:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443047424:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204189.569284:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443047552:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204189.569300:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204189.569308:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204189/real 1788204189] req@ffff90234519ed80 x1875064443047552/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204189.569315:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443047552/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204189.569320:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204189.569322:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204189.569324:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443047552:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204189.569327:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443047680:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204189.569337:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204189.569342:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204189/real 1788204189] req@ffff90234519d880 x1875064443047680/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204189.569348:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443047680/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204189.569353:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204189.569354:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204189.569356:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443047680:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204189.569359:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443047808:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204189.569367:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204189.569372:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204189/real 1788204189] req@ffff90234519e300 x1875064443047808/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204189.569378:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443047808/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204189.569382:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204189.569384:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204189.569385:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443047808:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204190.401036:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204191.425034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204192.449096:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204193.473081:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204194.497095:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204194.689034:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204194.689046:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204194.689051:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204194.689063:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3466 00000100:00080000:0.0:1788204194.689077:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204194.689085:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204194.689132:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443047936/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204194.689164:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204194.689168:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204194.689170:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204194.689175:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3466 00000100:00080000:0.0:1788204194.689178:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204194.689183:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204194.689195:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204194.689209:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064443048064/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204194.689218:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204194.689219:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443047936:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204194.689220:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204194.689223:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204194.689230:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3466 00000100:00080000:0.0:1788204194.689233:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204194.689236:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204194.689240:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204194.689251:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064443048192/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204194.689257:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204194.689260:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204194.689261:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204194.689263:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3466 00000100:00080000:0.0:1788204194.689266:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204194.689268:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204194.689271:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204194.689278:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064443048320/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204194.689301:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204194.689348:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204194/real 1788204194] req@ffff9023450bb480 x1875064443047936/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204194.689367:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443047936/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204194.689375:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204194.689378:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204194.689381:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443047936:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204194.689402:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443048064:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204194.689420:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204194.689428:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204194/real 1788204194] req@ffff9023450ba680 x1875064443048064/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204194.689436:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064443048064/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204194.689441:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204194.689443:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204194.689445:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443048064:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204194.689448:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443048192:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204194.689458:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204194.689463:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204194/real 1788204194] req@ffff9023450b8e00 x1875064443048192/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204194.689469:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064443048192/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204194.689474:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204194.689476:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204194.689477:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443048192:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204194.689481:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443048320:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204194.689490:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204194.689495:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204194/real 1788204194] req@ffff9023450bbb80 x1875064443048320/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204194.689502:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064443048320/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204194.689506:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204194.689508:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204194.689509:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443048320:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204195.521069:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204196.545042:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204197.569041:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204198.593024:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204199.617049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204199.808967:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204199.808977:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204199.808981:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204199.808991:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3471 00000100:00080000:0.0:1788204199.809001:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204199.809009:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204199.809050:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064443048448/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204199.809081:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204199.809085:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204199.809087:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204199.809091:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3471 00000100:00080000:0.0:1788204199.809095:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204199.809099:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204199.809108:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204199.809122:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064443048576/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204199.809138:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788204199.809141:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443048448:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204199.809142:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204199.809146:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204199.809152:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3471 00000100:00080000:0.0:1788204199.809156:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204199.809160:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204199.809164:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204199.809174:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064443048704/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204199.809182:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204199.809192:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204199.809194:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204199.809197:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3471 00000100:00080000:0.0:1788204199.809200:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788204199.809205:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788204199.809208:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204199.809214:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204199.809225:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443048832/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204199.809248:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204199/real 1788204199] req@ffff9023450bbb80 x1875064443048448/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204199.809266:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064443048448/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204199.809276:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204199.809279:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204199.809283:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443048448:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204199.809304:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443048576:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788204199.809326:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204199.809339:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204199/real 1788204199] req@ffff9023450b8e00 x1875064443048576/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204199.809351:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064443048576/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204199.809357:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204199.809359:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204199.809364:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443048576:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788204199.809370:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443048704:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204199.809390:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204199.809411:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204199/real 1788204199] req@ffff9023450ba680 x1875064443048704/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204199.809423:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064443048704/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204199.809431:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204199.809434:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204199.809438:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443048704:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788204199.809444:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443048832:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204199.809463:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204199.809499:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204199/real 1788204199] req@ffff9023450bb480 x1875064443048832/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204199.809511:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443048832/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204199.809519:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204199.809521:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204199.809523:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443048832:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204200.641029:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204201.665033:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204202.689096:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204203.713044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204204.737073:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204204.928983:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204204.928990:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204204.928994:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204204.929000:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3476 00000100:00080000:1.0:1788204204.929009:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204204.929013:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204204.929043:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064443048960/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204204.929062:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204204.929065:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204204.929066:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204204.929068:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3476 00000100:00080000:1.0:1788204204.929070:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204204.929072:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204204.929078:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204204.929087:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443049088/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204204.929093:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204204.929095:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204204.929096:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204204.929099:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3476 00000100:00080000:1.0:1788204204.929100:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204204.929102:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204204.929105:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204204.929109:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443048960:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204204.929113:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443049216/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204204.929120:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204204.929122:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204204.929123:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204204.929125:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3476 00000100:00080000:1.0:1788204204.929127:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204204.929128:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204204.929131:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204204.929136:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443049344/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204204.929178:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204204.929213:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204204/real 1788204204] req@ffff90234519f800 x1875064443048960/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204204.929230:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064443048960/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204204.929240:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204204.929244:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204204.929249:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443048960:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204204.929256:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443049088:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204204.929277:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204204.929289:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204204/real 1788204204] req@ffff90234519e300 x1875064443049088/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204204.929300:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443049088/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204204.929308:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204204.929310:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204204.929313:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443049088:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204204.929348:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443049216:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204204.929366:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204204.929377:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204204/real 1788204204] req@ffff90234519d880 x1875064443049216/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204204.929416:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443049216/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204204.929424:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204204.929428:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204204.929431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443049216:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204204.929436:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443049344:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204204.929454:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204204.929463:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204204/real 1788204204] req@ffff90234519ed80 x1875064443049344/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204204.929473:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443049344/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204204.929486:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204204.929489:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204204.929492:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443049344:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204205.761032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204206.785022:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204207.809088:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204208.832999:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204209.857019:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204210.048986:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204210.048994:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204210.048998:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204210.049005:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3481 00000100:00080000:2.0:1788204210.049014:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204210.049018:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204210.049053:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443049472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204210.049078:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204210.049082:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204210.049083:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204210.049087:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3481 00000100:00080000:2.0:1788204210.049091:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204210.049095:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204210.049104:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204210.049117:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443049472:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204210.049118:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443049600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204210.049132:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204210.049136:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204210.049138:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204210.049141:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3481 00000100:00080000:2.0:1788204210.049144:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204210.049148:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204210.049152:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204210.049162:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443049728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204210.049171:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788204210.049174:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204210.049178:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204210.049180:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204210.049185:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3481 00000100:00080000:2.0:1788204210.049189:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204210.049192:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204210.049197:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204210.049206:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204210/real 1788204210] req@ffff90234598e300 x1875064443049472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204210.049209:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443049856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204210.049232:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443049472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204210.049249:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204210.049254:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204210.049257:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443049472:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204210.049274:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443049600:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204210.049288:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204210.049323:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204210/real 1788204210] req@ffff90234598d180 x1875064443049600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204210.049331:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443049600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204210.049336:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204210.049338:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204210.049340:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443049600:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204210.049344:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443049728:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204210.049358:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204210.049364:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204210/real 1788204210] req@ffff90234598c380 x1875064443049728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204210.049371:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443049728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204210.049376:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204210.049378:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204210.049380:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443049728:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204210.049385:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443049856:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204210.049400:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204210.049410:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204210/real 1788204210] req@ffff90234598f480 x1875064443049856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204210.049421:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443049856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204210.049429:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204210.049432:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204210.049434:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443049856:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204210.881080:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204211.905035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204212.929041:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204213.953023:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204214.977041:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204215.168968:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204215.168978:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204215.168982:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204215.168992:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3486 00000100:00080000:2.0:1788204215.169002:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204215.169009:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204215.169045:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443049984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204215.169070:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204215.169073:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204215.169075:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204215.169079:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3486 00000100:00080000:2.0:1788204215.169083:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204215.169087:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204215.169096:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204215.169108:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443050112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204215.169116:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443049984:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204215.169120:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204215.169126:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204215.169128:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204215.169131:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3486 00000100:00080000:2.0:1788204215.169134:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204215.169138:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204215.169142:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204215.169154:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443050240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204215.169163:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204215.169167:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204215.169169:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204215.169172:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3486 00000100:00080000:2.0:1788204215.169176:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788204215.169179:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788204215.169184:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204215.169189:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204215.169201:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443050368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204215.169230:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204215/real 1788204215] req@ffff90234598ce00 x1875064443049984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204215.169249:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443049984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204215.169259:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204215.169263:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204215.169266:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443049984:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204215.169286:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443050112:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204215.169306:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204215.169318:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204215/real 1788204215] req@ffff90234598c700 x1875064443050112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204215.169337:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443050112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204215.169348:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204215.169351:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204215.169354:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443050112:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204215.169360:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443050240:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204215.169378:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204215.169389:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204215/real 1788204215] req@ffff90234598dc00 x1875064443050240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204215.169400:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443050240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204215.169418:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204215.169420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204215.169423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443050240:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204215.169435:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443050368:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204215.169451:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204215.169461:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204215/real 1788204215] req@ffff90234598f800 x1875064443050368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204215.169473:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443050368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204215.169480:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204215.169483:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204215.169486:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443050368:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204216.001002:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204217.025049:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204218.049024:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204219.073126:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204220.097044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204220.289008:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204220.289018:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204220.289022:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204220.289033:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3491 00000100:00080000:1.0:1788204220.289044:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204220.289051:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204220.289095:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443050496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204220.289132:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204220.289136:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204220.289137:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204220.289141:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3491 00000100:00080000:1.0:1788204220.289143:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204220.289146:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204220.289155:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204220.289166:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443050624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204220.289173:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204220.289176:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204220.289177:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204220.289180:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3491 00000100:00080000:1.0:1788204220.289184:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204220.289185:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443050496:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204220.289187:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204220.289193:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204220.289207:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443050752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204220.289213:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204220.289215:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204220.289217:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204220.289219:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3491 00000100:00080000:1.0:1788204220.289220:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204220.289223:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204220.289226:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204220.289232:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443050880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204220.289265:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204220.289312:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204220/real 1788204220] req@ffff90234519df80 x1875064443050496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204220.289327:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443050496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204220.289334:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204220.289337:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204220.289340:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443050496:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204220.289362:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443050624:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204220.289386:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204220.289398:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204220/real 1788204220] req@ffff90234519fb80 x1875064443050624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204220.289411:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443050624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204220.289419:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204220.289422:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204220.289426:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443050624:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204220.289430:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443050752:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204220.289456:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204220.289467:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204220/real 1788204220] req@ffff90234519c700 x1875064443050752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204220.289479:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443050752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204220.289486:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204220.289488:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204220.289490:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443050752:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204220.289494:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443050880:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204220.289507:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204220.289514:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204220/real 1788204220] req@ffff90234519c000 x1875064443050880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204220.289521:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443050880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204220.289526:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204220.289527:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204220.289529:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443050880:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204221.121088:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204222.145119:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204223.169059:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204224.193069:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204225.217049:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204225.409012:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204225.409023:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204225.409053:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204225.409063:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3496 00000100:00080000:1.0:1788204225.409073:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204225.409080:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204225.409116:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443051008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204225.409150:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204225.409154:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204225.409156:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204225.409161:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3496 00000100:00080000:1.0:1788204225.409164:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204225.409168:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204225.409177:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204225.409188:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443051136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204225.409196:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204225.409199:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204225.409201:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204225.409204:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3496 00000100:00080000:1.0:1788204225.409207:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204225.409210:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204225.409214:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204225.409223:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443051264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204225.409232:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204225.409235:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204225.409237:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204225.409240:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3496 00000100:00080000:1.0:1788204225.409243:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204225.409247:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204225.409251:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204225.409259:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443051392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204225.409373:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443051008:192.168.201.130@tcp:250:kworker.0 00000800:00000100:3.0:1788204225.409430:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204225.409468:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204225/real 1788204225] req@ffff90234519ea00 x1875064443051008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204225.409484:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443051008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204225.409493:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204225.409497:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204225.409500:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443051008:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204225.409507:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443051136:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204225.409528:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204225.409539:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204225/real 1788204225] req@ffff90234519f100 x1875064443051136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204225.409551:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443051136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204225.409563:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204225.409566:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204225.409569:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443051136:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204225.409575:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443051264:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204225.409592:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204225.409601:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204225/real 1788204225] req@ffff90234519c380 x1875064443051264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204225.409611:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443051264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204225.409618:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204225.409621:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204225.409624:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443051264:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204225.409639:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443051392:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204225.409659:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204225.409669:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204225/real 1788204225] req@ffff90234519f480 x1875064443051392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204225.409679:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443051392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204225.409686:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204225.409689:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204225.409692:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443051392:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204226.241024:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204227.265061:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204228.289035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204229.313073:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204230.337105:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204230.529007:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204230.529016:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204230.529019:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204230.529029:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3502 00000100:00080000:1.0:1788204230.529038:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204230.529044:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204230.529078:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064443051520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204230.529109:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204230.529120:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204230.529121:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204230.529125:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3502 00000100:00080000:1.0:1788204230.529129:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204230.529132:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204230.529141:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204230.529153:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443051648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204230.529171:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204230.529174:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204230.529176:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204230.529180:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3502 00000100:00080000:1.0:1788204230.529182:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204230.529185:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204230.529189:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204230.529197:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443051776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204230.529211:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204230.529214:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204230.529216:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204230.529219:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3502 00000100:00080000:1.0:1788204230.529221:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204230.529224:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204230.529228:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204230.529235:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443051904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204230.529341:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443051520:192.168.201.130@tcp:250:kworker.0 00000800:00000100:3.0:1788204230.529392:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204230.529425:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204230/real 1788204230] req@ffff90234519dc00 x1875064443051520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204230.529440:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064443051520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204230.529448:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204230.529451:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204230.529455:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443051520:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204230.529461:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443051648:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204230.529477:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204230.529487:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204230/real 1788204230] req@ffff90234519f480 x1875064443051648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204230.529504:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443051648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204230.529511:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204230.529514:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204230.529517:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443051648:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204230.529522:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443051776:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204230.529546:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204230.529555:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204230/real 1788204230] req@ffff90234519c380 x1875064443051776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204230.529565:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443051776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204230.529572:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204230.529574:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204230.529582:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443051776:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204230.529587:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443051904:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204230.529601:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204230.529609:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204230/real 1788204230] req@ffff90234519f100 x1875064443051904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204230.529618:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443051904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204230.529625:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204230.529628:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204230.529630:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443051904:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204231.361075:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204232.385093:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204233.409078:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204234.433077:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204235.457027:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204235.648997:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204235.649011:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204235.649016:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204235.649027:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3507 00000100:00080000:3.0:1788204235.649041:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204235.649050:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204235.649092:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443052032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204235.649119:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204235.649124:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204235.649126:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204235.649131:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3507 00000100:00080000:3.0:1788204235.649136:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204235.649142:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204235.649153:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204235.649157:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443052032:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204235.649175:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443052160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204235.649186:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204235.649189:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204235.649191:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204235.649195:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3507 00000100:00080000:3.0:1788204235.649198:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204235.649202:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204235.649207:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204235.649218:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443052288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204235.649226:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788204235.649267:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443052160:192.168.201.130@tcp:38:kworker.0 00000100:00080000:3.0:1788204235.649286:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204235.649289:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204235.649290:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443052288:192.168.201.130@tcp:8:kworker.0 00000100:00080000:3.0:1788204235.649299:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3507 00000100:00080000:3.0:1788204235.649305:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204235.649311:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204235.649317:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204235.649331:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443052416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204235.649363:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443052416:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204236.481021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204237.505296:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204238.529106:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788204238.721246:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:1.0:1788204238.721303:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788204238.721327:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204238.721382:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204238.721393:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204238.721403:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788204238.721447:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204235/real 1788204238] req@ffff902352689c00 x1875064443052416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204238.721481:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443052416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204238.721495:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204238.721502:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204238.721507:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443052416:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204238.721517:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204235/real 1788204238] req@ffff902352688e00 x1875064443052160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204238.721531:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443052160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204238.721541:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204238.721545:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204238.721548:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443052160:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204238.721557:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204235/real 1788204238] req@ffff90235268a680 x1875064443052288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204238.721571:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443052288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204238.721582:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204238.721586:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204238.721590:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443052288:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204238.721598:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204235/real 1788204238] req@ffff902352689500 x1875064443052032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204238.721613:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443052032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204238.721622:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204238.721626:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204238.721629:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443052032:192.168.201.130@tcp:250:kworker.0 00010000:00010000:0.0:1788204239.553647:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204240.577054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204240.769029:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204240.769039:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204240.769042:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204240.769050:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3512 00000100:00080000:0.0:1788204240.769059:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204240.769065:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204240.769100:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443052544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204240.769140:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204240.769144:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204240.769146:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204240.769150:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3512 00000100:00080000:0.0:1788204240.769153:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204240.769157:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204240.769166:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204240.769179:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443052672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204240.769187:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204240.769190:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204240.769192:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204240.769195:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3512 00000100:00080000:0.0:1788204240.769199:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204240.769202:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204240.769206:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204240.769215:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443052800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204240.769223:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204240.769227:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204240.769228:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204240.769232:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3512 00000100:00080000:0.0:1788204240.769235:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204240.769239:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204240.769242:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204240.769250:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443052928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204240.769338:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443052544:192.168.201.130@tcp:250:kworker.0 00000800:00000100:3.0:1788204240.769407:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204240.769451:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204240/real 1788204240] req@ffff9023450b8a80 x1875064443052544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204240.769472:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443052544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204240.769483:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204240.769488:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204240.769491:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443052544:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204240.769499:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443052672:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204240.769523:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204240.769535:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204240/real 1788204240] req@ffff9023450bb800 x1875064443052672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204240.769543:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443052672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204240.769549:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204240.769552:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204240.769554:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443052672:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204240.769558:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443052800:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204240.769571:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204240.769577:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204240/real 1788204240] req@ffff9023450b9500 x1875064443052800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204240.769585:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443052800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204240.769590:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204240.769592:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204240.769594:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443052800:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204240.769600:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443052928:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204240.769616:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204240.769626:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204240/real 1788204240] req@ffff9023450b8380 x1875064443052928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204240.769650:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443052928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204240.769658:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204240.769662:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204240.769665:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443052928:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204241.601026:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204242.625080:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204243.649007:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204244.673103:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204245.697011:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204245.888973:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204245.888981:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204245.888984:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204245.888991:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3517 00000100:00080000:0.0:1788204245.889000:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204245.889006:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204245.889050:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443053056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204245.889075:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204245.889078:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204245.889079:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204245.889082:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3517 00000100:00080000:0.0:1788204245.889085:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204245.889088:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204245.889096:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204245.889110:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443053184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204245.889117:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443053056:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204245.889126:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204245.889130:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204245.889132:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204245.889137:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3517 00000100:00080000:0.0:1788204245.889140:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204245.889143:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204245.889147:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204245.889157:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443053312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204245.889165:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204245.889168:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204245.889170:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204245.889173:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3517 00000100:00080000:0.0:1788204245.889177:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788204245.889183:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788204245.889184:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204245.889191:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204245.889204:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443053440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204245.889226:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204245/real 1788204245] req@ffff9023450b9c00 x1875064443053056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204245.889258:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443053056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204245.889267:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204245.889270:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204245.889273:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443053056:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204245.889294:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443053184:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204245.889315:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204245.889327:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204245/real 1788204245] req@ffff9023450b9f80 x1875064443053184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204245.889337:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443053184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204245.889344:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204245.889347:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204245.889350:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443053184:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204245.889355:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443053312:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204245.889373:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204245.889381:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204245/real 1788204245] req@ffff9023450b8700 x1875064443053312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204245.889392:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443053312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204245.889397:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204245.889399:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204245.889401:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443053312:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204245.889404:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443053440:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204245.889414:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204245.889420:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204245/real 1788204245] req@ffff9023450bad80 x1875064443053440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204245.889426:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443053440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204245.889430:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204245.889432:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204245.889433:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443053440:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204246.721027:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204247.745015:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204248.769062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204249.793038:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204250.817018:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204251.009000:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204251.009007:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204251.009010:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204251.009018:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3522 00000100:00080000:1.0:1788204251.009026:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204251.009031:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204251.009060:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443053568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204251.009080:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204251.009083:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204251.009084:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204251.009087:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3522 00000100:00080000:1.0:1788204251.009089:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204251.009091:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204251.009099:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204251.009107:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443053696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204251.009115:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204251.009118:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788204251.009118:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443053568:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204251.009121:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204251.009127:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3522 00000100:00080000:1.0:1788204251.009129:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204251.009132:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204251.009135:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204251.009142:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443053824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204251.009148:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204251.009151:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204251.009152:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204251.009154:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3522 00000100:00080000:1.0:1788204251.009157:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204251.009159:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204251.009161:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204251.009167:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443053952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204251.009182:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204251.009216:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204251/real 1788204251] req@ffff90234519ea00 x1875064443053568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204251.009234:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443053568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204251.009250:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204251.009253:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204251.009256:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443053568:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204251.009273:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443053696:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204251.009291:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204251.009301:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204251/real 1788204251] req@ffff90234519c000 x1875064443053696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204251.009309:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443053696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204251.009315:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204251.009318:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204251.009321:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443053696:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204251.009327:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443053824:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204251.009346:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204251.009358:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204251/real 1788204251] req@ffff90234519c700 x1875064443053824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204251.009371:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443053824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204251.009379:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204251.009381:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204251.009384:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443053824:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204251.009388:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443053952:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204251.009402:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204251.009408:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204251/real 1788204251] req@ffff90234519fb80 x1875064443053952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204251.009415:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443053952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204251.009420:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204251.009421:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204251.009423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443053952:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204251.841041:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204252.865034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204253.889062:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204254.913001:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204255.937027:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204256.128998:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204256.129005:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204256.129008:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204256.129016:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3527 00000100:00080000:1.0:1788204256.129023:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204256.129028:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204256.129057:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443054080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204256.129077:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204256.129081:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204256.129082:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204256.129086:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3527 00000100:00080000:1.0:1788204256.129088:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204256.129092:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204256.129100:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204256.129110:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443054208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204256.129118:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443054080:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204256.129119:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204256.129121:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204256.129123:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204256.129126:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3527 00000100:00080000:1.0:1788204256.129129:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204256.129131:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204256.129135:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204256.129143:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443054336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204256.129149:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204256.129152:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204256.129153:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204256.129156:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3527 00000100:00080000:1.0:1788204256.129159:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204256.129161:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204256.129164:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788204256.129171:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204256.129172:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443054464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204256.129206:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204256/real 1788204256] req@ffff90234519df80 x1875064443054080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204256.129222:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443054080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204256.129229:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204256.129231:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204256.129233:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443054080:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204256.129250:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443054208:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204256.129265:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204256.129273:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204256/real 1788204256] req@ffff90234519ed80 x1875064443054208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204256.129280:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443054208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204256.129285:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204256.129287:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204256.129288:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443054208:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204256.129291:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443054336:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204256.129300:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204256.129305:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204256/real 1788204256] req@ffff90234519d880 x1875064443054336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204256.129312:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443054336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204256.129316:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204256.129317:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204256.129319:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443054336:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204256.129322:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443054464:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204256.129337:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204256.129345:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204256/real 1788204256] req@ffff90234519e300 x1875064443054464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204256.129351:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443054464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204256.129360:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204256.129362:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204256.129364:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443054464:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204256.961027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204257.985046:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204259.009036:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204260.033029:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204261.057062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204261.248994:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204261.249012:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204261.249019:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204261.249027:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3532 00000100:00080000:3.0:1788204261.249038:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204261.249044:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204261.249079:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443054592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204261.249108:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204261.249112:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204261.249113:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204261.249117:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3532 00000100:00080000:3.0:1788204261.249120:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204261.249124:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204261.249133:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204261.249146:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443054720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204261.249149:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443054592:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204261.249166:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204261.249170:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204261.249172:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204261.249177:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3532 00000100:00080000:3.0:1788204261.249180:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204261.249184:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204261.249189:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204261.249198:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443054848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204261.249207:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204261.249211:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1788204261.249212:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204261.249214:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204261.249220:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3532 00000100:00080000:3.0:1788204261.249224:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204261.249228:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204261.249233:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204261.249244:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443054976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204261.249253:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204261/real 1788204261] req@ffff902352689180 x1875064443054592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204261.249274:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443054592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204261.249284:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204261.249288:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204261.249291:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443054592:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204261.249312:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443054720:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204261.249333:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204261.249345:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204261/real 1788204261] req@ffff902352688700 x1875064443054720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204261.249371:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443054720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204261.249379:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204261.249381:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204261.249384:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443054720:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204261.249390:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443054848:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204261.249427:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204261.249438:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204261/real 1788204261] req@ffff90235268bb80 x1875064443054848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204261.249448:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443054848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204261.249461:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204261.249463:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204261.249466:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443054848:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204261.249472:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443054976:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204261.249490:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204261.249500:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204261/real 1788204261] req@ffff902352689880 x1875064443054976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204261.249509:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443054976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204261.249516:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204261.249519:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204261.249522:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443054976:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204262.081030:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204263.105007:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204264.129054:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204265.153013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204266.177039:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204266.368965:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204266.368972:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204266.368976:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204266.368982:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3537 00000100:00080000:3.0:1788204266.368991:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204266.368996:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204266.369025:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443055104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204266.369051:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204266.369053:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204266.369055:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204266.369057:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3537 00000100:00080000:3.0:1788204266.369059:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204266.369061:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204266.369068:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204266.369076:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443055232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204266.369082:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204266.369085:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204266.369086:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204266.369088:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3537 00000100:00080000:3.0:1788204266.369089:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204266.369091:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204266.369094:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204266.369103:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443055104:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204266.369104:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443055360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204266.369119:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204266.369122:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204266.369124:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204266.369127:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3537 00000100:00080000:3.0:1788204266.369131:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204266.369134:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204266.369139:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204266.369149:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443055488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204266.369168:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204266.369198:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204266/real 1788204266] req@ffff902352688380 x1875064443055104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204266.369209:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443055104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204266.369216:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204266.369219:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204266.369222:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443055104:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204266.369227:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443055232:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204266.369241:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204266.369248:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204266/real 1788204266] req@ffff90235268aa00 x1875064443055232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204266.369254:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443055232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204266.369259:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204266.369260:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204266.369262:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443055232:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204266.369278:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443055360:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204266.369288:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204266.369293:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204266/real 1788204266] req@ffff90235268ad80 x1875064443055360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204266.369302:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443055360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204266.369309:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204266.369312:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204266.369314:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443055360:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204266.369350:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443055488:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204266.369371:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204266.369379:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204266/real 1788204266] req@ffff90235268b480 x1875064443055488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204266.369387:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443055488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204266.369391:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204266.369393:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204266.369395:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443055488:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204267.201002:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204268.225027:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204269.248991:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204270.273033:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204271.297024:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204271.488973:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204271.488980:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204271.488984:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204271.488992:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3543 00000100:00080000:0.0:1788204271.489000:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204271.489008:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204271.489041:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443055616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204271.489066:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204271.489070:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204271.489072:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204271.489076:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3543 00000100:00080000:0.0:1788204271.489081:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204271.489086:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204271.489094:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204271.489099:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443055616:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204271.489109:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443055744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204271.489119:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204271.489121:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204271.489123:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204271.489126:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3543 00000100:00080000:0.0:1788204271.489129:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204271.489133:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204271.489136:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204271.489145:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443055872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204271.489151:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788204271.489154:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788204271.489155:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204271.489158:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204271.489161:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3543 00000100:00080000:0.0:1788204271.489164:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204271.489166:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204271.489170:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204271.489177:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443056000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204271.489189:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204271/real 1788204271] req@ffff9023450baa00 x1875064443055616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204271.489203:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443055616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204271.489210:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204271.489212:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204271.489215:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443055616:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204271.489248:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443055744:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204271.489270:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204271.489283:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204271/real 1788204271] req@ffff9023450b9180 x1875064443055744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204271.489292:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443055744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204271.489297:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204271.489299:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204271.489301:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443055744:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204271.489305:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443055872:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204271.489326:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204271.489333:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204271/real 1788204271] req@ffff9023450b8000 x1875064443055872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204271.489340:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443055872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204271.489344:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204271.489346:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204271.489355:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443055872:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204271.489358:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443056000:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204271.489369:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204271.489375:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204271/real 1788204271] req@ffff9023450ba300 x1875064443056000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204271.489381:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443056000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204271.489386:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204271.489388:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204271.489391:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443056000:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204272.321015:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204273.345029:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204274.369015:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204275.393024:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204276.417045:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204276.608984:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204276.608992:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204276.608995:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204276.609003:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3548 00000100:00080000:3.0:1788204276.609011:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204276.609016:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204276.609046:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064443056128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204276.609068:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204276.609071:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204276.609072:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204276.609075:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3548 00000100:00080000:3.0:1788204276.609076:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204276.609078:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204276.609085:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204276.609094:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443056256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204276.609102:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788204276.609103:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443056128:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204276.609108:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204276.609112:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204276.609116:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3548 00000100:00080000:3.0:1788204276.609117:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204276.609120:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204276.609123:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204276.609137:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443056384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204276.609160:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204276.609162:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204276.609164:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204276.609166:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3548 00000100:00080000:3.0:1788204276.609169:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788204276.609171:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204276.609174:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204276.609179:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204276.609189:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443056512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204276.609208:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204276/real 1788204276] req@ffff902352688a80 x1875064443056128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204276.609221:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064443056128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204276.609228:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204276.609232:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204276.609235:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443056128:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204276.609257:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443056256:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204276.609278:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204276.609289:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204276/real 1788204276] req@ffff90235268b480 x1875064443056256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204276.609299:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443056256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204276.609307:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204276.609309:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204276.609312:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443056256:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204276.609318:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443056384:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204276.609332:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204276.609356:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204276/real 1788204276] req@ffff90235268ad80 x1875064443056384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204276.609371:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443056384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204276.609379:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204276.609381:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204276.609384:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443056384:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204276.609390:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443056512:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204276.609409:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204276.609419:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204276/real 1788204276] req@ffff90235268aa00 x1875064443056512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204276.609429:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443056512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204276.609436:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204276.609439:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204276.609442:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443056512:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204277.441044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204278.465024:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204279.489007:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204280.513038:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204281.537044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204281.728994:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204281.729004:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204281.729008:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204281.729018:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3553 00000100:00080000:3.0:1788204281.729030:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204281.729079:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204281.729118:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443056640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204281.729152:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204281.729155:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204281.729157:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204281.729162:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3553 00000100:00080000:3.0:1788204281.729165:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204281.729169:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204281.729179:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204281.729192:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443056768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204281.729212:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443056640:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204281.729213:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204281.729222:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204281.729225:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204281.729229:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3553 00000100:00080000:3.0:1788204281.729233:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204281.729238:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204281.729244:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204281.729257:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443056896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204281.729267:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204281.729271:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204281.729273:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204281.729277:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3553 00000100:00080000:3.0:1788204281.729288:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788204281.729292:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204281.729296:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204281.729301:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204281.729313:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443057024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204281.729340:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204281/real 1788204281] req@ffff902352688380 x1875064443056640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204281.729358:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443056640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204281.729367:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204281.729371:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204281.729374:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443056640:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204281.729397:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443056768:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204281.729418:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204281.729429:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204281/real 1788204281] req@ffff902352689880 x1875064443056768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204281.729439:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443056768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204281.729446:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204281.729449:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204281.729452:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443056768:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204281.729457:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443056896:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204281.729473:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204281.729483:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204281/real 1788204281] req@ffff90235268bb80 x1875064443056896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204281.729499:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443056896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204281.729507:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204281.729510:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204281.729512:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443056896:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204281.729518:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443057024:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204281.729536:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204281.729546:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204281/real 1788204281] req@ffff902352688700 x1875064443057024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204281.729555:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443057024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204281.729562:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204281.729565:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204281.729567:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443057024:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204282.561032:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204283.585015:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204284.609055:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204285.633023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204286.657032:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204286.849030:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204286.849042:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204286.849048:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204286.849059:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3558 00000100:00080000:3.0:1788204286.849072:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204286.849079:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204286.849123:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443057152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204286.849154:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204286.849157:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204286.849159:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204286.849162:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3558 00000100:00080000:3.0:1788204286.849166:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204286.849169:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204286.849179:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204286.849193:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443057280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204286.849200:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204286.849204:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204286.849205:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204286.849209:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3558 00000100:00100000:0.0:1788204286.849214:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443057152:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204286.849215:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204286.849224:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204286.849230:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204286.849242:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443057408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204286.849258:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204286.849262:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204286.849264:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204286.849268:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3558 00000100:00080000:3.0:1788204286.849271:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204286.849275:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204286.849280:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204286.849292:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443057536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204286.849301:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204286.849360:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204286/real 1788204286] req@ffff902352689180 x1875064443057152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204286.849379:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443057152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204286.849387:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204286.849390:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204286.849393:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443057152:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204286.849398:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443057280:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204286.849417:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204286.849426:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204286/real 1788204286] req@ffff902352689500 x1875064443057280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204286.849433:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443057280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204286.849438:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204286.849440:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204286.849441:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443057280:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204286.849460:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443057408:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204286.849470:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204286.849476:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204286/real 1788204286] req@ffff90235268a680 x1875064443057408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204286.849482:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443057408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204286.849487:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204286.849489:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204286.849490:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443057408:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204286.849493:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443057536:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204286.849502:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204286.849507:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204286/real 1788204286] req@ffff902352688e00 x1875064443057536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204286.849513:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443057536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204286.849518:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204286.849520:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204286.849522:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443057536:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204287.681179:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204288.705125:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204289.729040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204290.753081:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204291.777003:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204291.969005:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204291.969013:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204291.969016:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204291.969024:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3563 00000100:00080000:0.0:1788204291.969032:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204291.969037:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204291.969070:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443057664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204291.969091:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204291.969093:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204291.969095:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204291.969097:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3563 00000100:00080000:0.0:1788204291.969100:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204291.969102:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204291.969116:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204291.969121:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443057664:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204291.969128:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443057792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204291.969135:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204291.969138:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204291.969139:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204291.969141:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3563 00000100:00080000:0.0:1788204291.969144:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204291.969146:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204291.969149:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204291.969156:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443057920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204291.969162:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204291.969164:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204291.969165:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204291.969167:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3563 00000100:00080000:0.0:1788204291.969170:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204291.969172:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204291.969174:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788204291.969180:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204291.969183:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443058048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204291.969219:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204291/real 1788204291] req@ffff9023450ba300 x1875064443057664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204291.969233:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443057664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204291.969240:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204291.969242:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204291.969245:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443057664:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204291.969261:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443057792:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204291.969274:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204291.969281:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204291/real 1788204291] req@ffff9023450b8000 x1875064443057792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204291.969289:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443057792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204291.969294:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204291.969296:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204291.969298:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443057792:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204291.969301:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443057920:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204291.969311:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204291.969316:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204291/real 1788204291] req@ffff9023450b9180 x1875064443057920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204291.969323:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443057920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204291.969328:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204291.969329:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204291.969331:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443057920:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204291.969334:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443058048:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204291.969343:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204291.969348:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204291/real 1788204291] req@ffff9023450baa00 x1875064443058048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204291.969354:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443058048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204291.969362:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204291.969364:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204291.969365:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443058048:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204292.801034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204293.825071:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204294.849080:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204295.873010:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204296.897043:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204297.088983:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204297.088993:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204297.088996:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204297.089003:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3568 00000100:00080000:3.0:1788204297.089012:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204297.089016:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204297.089045:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443058176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204297.089065:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204297.089067:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204297.089069:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204297.089071:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3568 00000100:00080000:3.0:1788204297.089073:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204297.089075:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204297.089083:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204297.089096:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443058304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204297.089100:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443058176:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204297.089113:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204297.089117:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204297.089119:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204297.089124:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3568 00000100:00080000:3.0:1788204297.089127:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204297.089130:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204297.089135:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204297.089144:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443058432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204297.089149:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204297.089152:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204297.089153:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204297.089155:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3568 00000100:00080000:3.0:1788204297.089158:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204297.089160:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204297.089163:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204297.089167:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204297.089172:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064443058560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204297.089199:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204297/real 1788204297] req@ffff902352689c00 x1875064443058176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204297.089215:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443058176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204297.089222:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204297.089225:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204297.089227:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443058176:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204297.089244:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443058304:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204297.089258:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204297.089266:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204297/real 1788204297] req@ffff90235268b800 x1875064443058304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204297.089272:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443058304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204297.089277:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204297.089278:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204297.089280:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443058304:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204297.089283:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443058432:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204297.089293:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204297.089298:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204297/real 1788204297] req@ffff90235268a300 x1875064443058432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204297.089304:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443058432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204297.089313:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204297.089315:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204297.089317:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443058432:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204297.089320:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443058560:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204297.089327:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204297.089343:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204297/real 1788204297] req@ffff90235268b100 x1875064443058560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204297.089350:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064443058560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204297.089354:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204297.089356:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204297.089358:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443058560:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204297.921026:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204298.945032:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204299.969025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204300.993014:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204302.017008:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204302.208950:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204302.208960:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204302.208963:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204302.208973:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3573 00000100:00080000:3.0:1788204302.208983:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204302.208991:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204302.209020:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443058688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204302.209043:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204302.209045:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204302.209047:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204302.209049:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3573 00000100:00080000:3.0:1788204302.209051:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204302.209053:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204302.209059:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204302.209069:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064443058816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204302.209076:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204302.209078:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204302.209079:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204302.209083:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3573 00000100:00100000:0.0:1788204302.209085:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443058688:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204302.209085:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204302.209093:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204302.209097:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204302.209104:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443058944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204302.209109:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204302.209112:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204302.209113:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204302.209115:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3573 00000100:00080000:3.0:1788204302.209118:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204302.209120:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204302.209123:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204302.209129:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443059072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204302.209184:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443058816:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204302.209209:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443058944:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204302.209221:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443059072:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204303.041149:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204304.065044:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204305.089429:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788204305.281091:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788204305.281123:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788204305.281138:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788204305.281173:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788204305.281181:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788204305.281189:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788204305.281217:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204302/real 1788204305] req@ffff90235268b100 x1875064443058816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204305.281247:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064443058816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204305.281259:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204305.281263:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204305.281267:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443058816:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204305.281274:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204302/real 1788204305] req@ffff90235268a300 x1875064443058944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204305.281286:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443058944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204305.281294:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204305.281297:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204305.281300:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443058944:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204305.281306:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204302/real 1788204305] req@ffff90235268b800 x1875064443059072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204305.281317:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443059072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204305.281324:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204305.281327:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204305.281331:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443059072:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204305.281337:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204302/real 1788204305] req@ffff902352688000 x1875064443058688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204305.281348:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443058688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204305.281356:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204305.281359:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204305.281362:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443058688:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788204306.113076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204307.137060:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204307.329049:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204307.329070:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204307.329073:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204307.329081:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3578 00000100:00080000:2.0:1788204307.329089:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204307.329096:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204307.329131:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064443059200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204307.329157:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204307.329161:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204307.329163:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204307.329168:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3578 00000100:00080000:2.0:1788204307.329172:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204307.329177:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204307.329188:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204307.329202:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064443059328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204307.329212:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204307.329215:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204307.329217:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204307.329220:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3578 00000100:00080000:2.0:1788204307.329223:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204307.329226:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204307.329231:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204307.329241:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064443059456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204307.329250:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204307.329252:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204307.329254:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204307.329257:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3578 00000100:00080000:2.0:1788204307.329260:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204307.329263:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204307.329267:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204307.329277:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443059584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204307.329291:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443059200:192.168.201.130@tcp:250:kworker.0 00000800:00000100:0.0:1788204307.329363:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204307.329408:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204307/real 1788204307] req@ffff90234598d880 x1875064443059200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204307.329427:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064443059200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204307.329435:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204307.329440:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204307.329443:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443059200:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204307.329449:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443059328:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204307.329471:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204307.329483:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204307/real 1788204307] req@ffff90234598e680 x1875064443059328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204307.329493:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064443059328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204307.329499:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204307.329502:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204307.329505:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443059328:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204307.329510:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443059456:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204307.329526:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204307.329536:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204307/real 1788204307] req@ffff90234598ca80 x1875064443059456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204307.329545:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064443059456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204307.329553:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204307.329555:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204307.329558:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443059456:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204307.329584:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443059584:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204307.329602:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204307.329632:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204307/real 1788204307] req@ffff90234598f800 x1875064443059584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204307.329644:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443059584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204307.329650:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204307.329653:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204307.329656:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443059584:192.168.201.130@tcp:8:kworker.0 00010000:00010000:3.0:1788204308.160982:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204309.185034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204310.208995:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204311.232996:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204312.256982:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204312.448990:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204312.449000:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204312.449004:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204312.449013:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3583 00000100:00080000:2.0:1788204312.449023:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204312.449029:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204312.449065:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443059712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204312.449088:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204312.449091:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204312.449093:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204312.449097:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3583 00000100:00080000:2.0:1788204312.449101:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204312.449106:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204312.449115:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204312.449129:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443059712:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204312.449135:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443059840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204312.449143:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204312.449145:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204312.449147:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204312.449149:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3583 00000100:00080000:2.0:1788204312.449152:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204312.449155:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204312.449158:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204312.449165:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443059968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204312.449173:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204312.449175:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204312.449176:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204312.449178:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3583 00000100:00080000:2.0:1788204312.449180:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788204312.449184:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788204312.449185:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204312.449190:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204312.449196:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443060096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204312.449222:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204312/real 1788204312] req@ffff90234598dc00 x1875064443059712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204312.449240:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443059712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204312.449250:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204312.449254:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204312.449266:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443059712:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204312.449288:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443059840:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204312.449313:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204312.449323:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204312/real 1788204312] req@ffff90234598c700 x1875064443059840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204312.449330:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443059840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204312.449336:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204312.449338:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204312.449339:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443059840:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204312.449343:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443059968:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204312.449355:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204312.449360:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204312/real 1788204312] req@ffff90234598ce00 x1875064443059968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204312.449371:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443059968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204312.449375:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204312.449377:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204312.449379:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443059968:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204312.449382:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443060096:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204312.449390:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204312.449395:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204312/real 1788204312] req@ffff90234598f480 x1875064443060096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204312.449401:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443060096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204312.449406:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204312.449407:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204312.449409:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443060096:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204313.281005:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204314.305088:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204315.329074:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204316.353036:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204317.377092:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204317.569002:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204317.569012:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204317.569016:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204317.569025:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3589 00000100:00080000:1.0:1788204317.569036:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204317.569043:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204317.569083:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064443060224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204317.569107:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204317.569110:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204317.569111:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204317.569114:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3589 00000100:00080000:1.0:1788204317.569116:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204317.569119:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204317.569127:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204317.569139:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443060352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204317.569146:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443060224:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204317.569149:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204317.569154:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204317.569156:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204317.569159:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3589 00000100:00080000:1.0:1788204317.569161:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204317.569165:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204317.569167:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204317.569175:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443060480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204317.569180:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204317.569182:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204317.569183:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204317.569186:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3589 00000100:00080000:1.0:1788204317.569189:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204317.569191:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204317.569194:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204317.569200:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443060608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204317.569220:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204317.569268:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204317/real 1788204317] req@ffff90234519f800 x1875064443060224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204317.569282:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064443060224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204317.569296:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204317.569300:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204317.569303:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443060224:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204317.569323:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443060352:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204317.569337:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204317.569345:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204317/real 1788204317] req@ffff90234519e300 x1875064443060352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204317.569353:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443060352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204317.569358:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204317.569360:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204317.569362:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443060352:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204317.569365:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443060480:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204317.569375:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204317.569380:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204317/real 1788204317] req@ffff90234519d880 x1875064443060480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204317.569387:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443060480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204317.569392:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204317.569394:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204317.569396:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443060480:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204317.569399:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443060608:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204317.569409:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204317.569414:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204317/real 1788204317] req@ffff90234519ed80 x1875064443060608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204317.569420:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443060608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204317.569425:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204317.569426:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204317.569453:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443060608:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204318.401073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204319.425014:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204320.449039:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204321.473011:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204322.497051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204322.689040:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204322.689049:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204322.689053:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204322.689063:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3594 00000100:00080000:0.0:1788204322.689076:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204322.689084:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204322.689131:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443060736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204322.689163:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204322.689168:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204322.689170:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204322.689174:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3594 00000100:00080000:0.0:1788204322.689179:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204322.689185:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204322.689197:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204322.689211:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443060864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204322.689220:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204322.689224:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204322.689226:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204322.689232:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3594 00000100:00100000:3.0:1788204322.689234:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443060736:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204322.689236:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204322.689246:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204322.689252:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204322.689264:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443060992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204322.689270:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204322.689273:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204322.689274:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204322.689276:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3594 00000100:00080000:0.0:1788204322.689278:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204322.689282:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204322.689286:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204322.689296:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443061120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204322.689305:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204322.689362:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204322/real 1788204322] req@ffff9023450bad80 x1875064443060736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204322.689382:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443060736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204322.689391:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204322.689394:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204322.689397:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443060736:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204322.689417:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443060864:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204322.689438:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204322.689449:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204322/real 1788204322] req@ffff9023450b8700 x1875064443060864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204322.689467:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443060864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204322.689474:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204322.689477:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204322.689479:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443060864:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204322.689483:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443060992:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204322.689498:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204322.689505:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204322/real 1788204322] req@ffff9023450b9f80 x1875064443060992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204322.689512:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443060992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204322.689517:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204322.689519:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204322.689520:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443060992:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204322.689524:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443061120:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204322.689532:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204322.689537:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204322/real 1788204322] req@ffff9023450b9c00 x1875064443061120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204322.689544:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443061120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204322.689562:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204322.689565:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204322.689568:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443061120:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204323.521056:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204324.545079:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204325.569056:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204326.593076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204327.617146:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204327.809049:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204327.809059:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204327.809064:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204327.809076:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3599 00000100:00080000:2.0:1788204327.809090:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204327.809097:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204327.809149:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443061248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204327.809192:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204327.809197:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204327.809199:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204327.809205:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3599 00000100:00080000:2.0:1788204327.809209:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204327.809214:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204327.809227:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204327.809241:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443061376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204327.809248:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443061248:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204327.809271:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204327.809282:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204327.809284:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204327.809288:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3599 00000100:00080000:2.0:1788204327.809291:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204327.809295:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204327.809299:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204327.809311:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443061504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204327.809319:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204327.809323:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204327.809324:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204327.809326:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3599 00000100:00080000:2.0:1788204327.809337:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204327.809340:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204327.809343:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788204327.809350:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204327.809355:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443061632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204327.809396:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204327/real 1788204327] req@ffff90234598c380 x1875064443061248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204327.809412:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443061248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204327.809421:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204327.809424:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204327.809427:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443061248:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204327.809452:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443061376:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204327.809468:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204327.809477:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204327/real 1788204327] req@ffff90234598d180 x1875064443061376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204327.809486:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443061376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204327.809491:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204327.809493:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204327.809496:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443061376:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204327.809500:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443061504:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204327.809512:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204327.809518:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204327/real 1788204327] req@ffff90234598e300 x1875064443061504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204327.809525:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443061504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204327.809531:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204327.809533:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204327.809535:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443061504:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204327.809539:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443061632:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204327.809557:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204327.809569:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204327/real 1788204327] req@ffff90234598df80 x1875064443061632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204327.809581:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443061632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204327.809589:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204327.809591:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204327.809594:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443061632:192.168.201.130@tcp:8:kworker.0 00010000:00010000:3.0:1788204328.641030:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204329.665110:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204330.689057:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204331.713103:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204332.737036:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204332.928997:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204332.929005:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204332.929009:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204332.929017:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3604 00000100:00080000:2.0:1788204332.929025:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204332.929031:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204332.929059:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064443061760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204332.929080:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204332.929082:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204332.929083:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204332.929086:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3604 00000100:00080000:2.0:1788204332.929088:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204332.929090:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204332.929097:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204332.929105:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064443061888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204332.929113:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204332.929115:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204332.929116:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204332.929119:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3604 00000100:00080000:2.0:1788204332.929120:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204332.929122:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204332.929124:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204332.929130:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064443062016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204332.929144:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204332.929151:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204332.929154:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788204332.929154:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443061760:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204332.929190:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3604 00000100:00080000:2.0:1788204332.929194:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204332.929197:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204332.929203:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204332.929217:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443062144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204332.929268:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204332.929332:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204332/real 1788204332] req@ffff90234598f100 x1875064443061760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204332.929354:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064443061760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204332.929367:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204332.929371:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204332.929376:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443061760:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204332.929385:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443061888:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204332.929410:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204332.929425:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204332/real 1788204332] req@ffff90234598ed80 x1875064443061888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204332.929438:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064443061888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204332.929446:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204332.929450:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204332.929454:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443061888:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204332.929485:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443062016:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204332.929507:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204332.929520:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204332/real 1788204332] req@ffff90234598fb80 x1875064443062016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204332.929540:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064443062016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204332.929549:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204332.929552:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204332.929555:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443062016:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204332.929562:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443062144:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204332.929582:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204332.929593:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204332/real 1788204332] req@ffff90234598df80 x1875064443062144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204332.929605:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443062144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204332.929614:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204332.929618:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204332.929621:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443062144:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204333.761098:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204334.785054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204335.809029:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204336.832980:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204337.857043:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204338.048971:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204338.048979:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204338.048982:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204338.048989:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3609 00000100:00080000:2.0:1788204338.048997:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204338.049002:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204338.049035:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443062272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204338.049060:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204338.049063:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204338.049064:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204338.049085:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3609 00000100:00080000:2.0:1788204338.049089:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204338.049092:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204338.049102:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204338.049116:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443062400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204338.049120:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443062272:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204338.049132:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204338.049136:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204338.049137:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204338.049141:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3609 00000100:00080000:2.0:1788204338.049145:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204338.049148:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204338.049153:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204338.049163:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443062528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204338.049171:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204338.049174:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204338.049176:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204338.049179:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3609 00000100:00080000:2.0:1788204338.049182:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204338.049185:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:3.0:1788204338.049190:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:2.0:1788204338.049192:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204338.049203:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443062656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204338.049235:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204338/real 1788204338] req@ffff90234598e300 x1875064443062272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204338.049253:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443062272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204338.049264:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204338.049267:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204338.049271:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443062272:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204338.049292:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443062400:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204338.049310:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204338.049321:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204338/real 1788204338] req@ffff90234598d180 x1875064443062400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204338.049331:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443062400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204338.049339:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204338.049342:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204338.049345:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443062400:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204338.049350:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443062528:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204338.049366:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204338.049374:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204338/real 1788204338] req@ffff90234598c380 x1875064443062528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204338.049385:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443062528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204338.049392:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204338.049395:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204338.049407:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443062528:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204338.049414:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443062656:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204338.049429:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204338.049438:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204338/real 1788204338] req@ffff90234598f480 x1875064443062656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204338.049455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443062656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204338.049461:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204338.049464:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204338.049466:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443062656:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204338.881015:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204339.905018:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204340.929013:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204341.954259:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204342.977089:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204343.168971:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204343.168979:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204343.168994:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204343.169003:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3614 00000100:00080000:2.0:1788204343.169011:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204343.169016:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204343.169043:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443062784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204343.169062:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204343.169065:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204343.169066:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204343.169070:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3614 00000100:00080000:2.0:1788204343.169072:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204343.169075:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204343.169082:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204343.169091:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443062912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204343.169098:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204343.169101:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788204343.169101:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443062784:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204343.169103:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204343.169109:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3614 00000100:00080000:2.0:1788204343.169113:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204343.169115:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204343.169119:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204343.169127:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443063040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204343.169137:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204343.169139:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204343.169140:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204343.169143:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3614 00000100:00080000:2.0:1788204343.169157:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204343.169160:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204343.169164:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788204343.169168:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204343.169173:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443063168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204343.169209:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204343/real 1788204343] req@ffff90234598ce00 x1875064443062784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204343.169265:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443062784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204343.169273:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204343.169276:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204343.169280:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443062784:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204343.169305:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443062912:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204343.169328:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204343.169435:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204343/real 1788204343] req@ffff90234598c700 x1875064443062912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204343.169449:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443062912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204343.169458:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204343.169463:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204343.169467:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443062912:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204343.169514:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443063040:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204343.169537:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204343.169549:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204343/real 1788204343] req@ffff90234598dc00 x1875064443063040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204343.169561:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443063040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204343.169569:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204343.169571:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204343.169574:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443063040:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204343.169580:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443063168:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204343.169598:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204343.169606:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204343/real 1788204343] req@ffff90234598f800 x1875064443063168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204343.169613:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443063168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204343.169618:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204343.169620:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204343.169622:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443063168:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204344.001025:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204345.025004:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204346.049049:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204347.073045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204348.097042:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204348.289013:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204348.289024:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204348.289028:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204348.289037:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3619 00000100:00080000:0.0:1788204348.289046:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204348.289053:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204348.289098:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443063296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204348.289124:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204348.289127:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204348.289129:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204348.289132:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3619 00000100:00080000:0.0:1788204348.289136:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204348.289139:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204348.289148:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204348.289161:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443063424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204348.289172:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204348.289176:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204348.289177:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204348.289180:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3619 00000100:00100000:3.0:1788204348.289180:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443063296:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204348.289186:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204348.289192:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204348.289197:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204348.289210:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443063552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204348.289221:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204348.289224:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204348.289226:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788204348.289246:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788204348.289253:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3619 00000100:00080000:0.0:1788204348.289260:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204348.289266:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204348.289272:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204348.289283:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443063680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204348.289294:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204348/real 1788204348] req@ffff9023450b8380 x1875064443063296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204348.289315:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443063296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204348.289325:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204348.289329:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204348.289332:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443063296:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204348.289354:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443063424:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204348.289373:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204348.289382:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204348/real 1788204348] req@ffff9023450b9500 x1875064443063424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204348.289389:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443063424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204348.289395:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204348.289396:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204348.289399:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443063424:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204348.289403:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443063552:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204348.289412:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204348.289418:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204348/real 1788204348] req@ffff9023450bb800 x1875064443063552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204348.289424:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443063552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204348.289429:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204348.289431:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204348.289432:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443063552:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204348.289435:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443063680:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204348.289444:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204348.289449:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204348/real 1788204348] req@ffff9023450b8a80 x1875064443063680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204348.289455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443063680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204348.289460:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204348.289461:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204348.289463:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443063680:192.168.201.130@tcp:8:kworker.0 00010000:00010000:3.0:1788204349.121111:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204350.145030:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204351.169035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204352.193035:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204353.216998:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204353.409000:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204353.409009:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204353.409013:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204353.409020:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3624 00000100:00080000:1.0:1788204353.409029:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204353.409035:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204353.409073:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443063808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204353.409098:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204353.409101:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204353.409102:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204353.409106:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3624 00000100:00080000:1.0:1788204353.409109:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204353.409112:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204353.409120:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204353.409131:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443063936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204353.409141:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204353.409144:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204353.409146:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788204353.409149:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443063808:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204353.409152:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3624 00000100:00080000:1.0:1788204353.409157:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204353.409161:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204353.409165:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204353.409175:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443064064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204353.409183:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204353.409186:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204353.409187:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204353.409190:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3624 00000100:00080000:1.0:1788204353.409193:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204353.409197:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204353.409201:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204353.409209:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443064192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204353.409222:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204353.409270:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204353/real 1788204353] req@ffff90234519df80 x1875064443063808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204353.409316:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443063808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204353.409326:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204353.409330:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204353.409334:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443063808:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204353.409355:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443063936:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204353.409377:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204353.409390:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204353/real 1788204353] req@ffff90234519fb80 x1875064443063936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204353.409402:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443063936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204353.409409:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204353.409412:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204353.409416:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443063936:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204353.409422:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443064064:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204353.409439:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204353.409459:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204353/real 1788204353] req@ffff90234519c700 x1875064443064064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204353.409475:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443064064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204353.409483:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204353.409486:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204353.409489:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443064064:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204353.409494:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443064192:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204353.409512:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204353.409522:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204353/real 1788204353] req@ffff90234519c000 x1875064443064192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204353.409533:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443064192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204353.409540:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204353.409542:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204353.409545:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443064192:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204354.240986:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204355.265048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204356.289028:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204357.312989:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204358.337129:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204358.528968:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204358.528974:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204358.528978:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204358.528985:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3630 00000100:00080000:2.0:1788204358.528994:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204358.528999:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204358.529028:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064443064320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204358.529047:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204358.529049:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204358.529050:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204358.529053:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3630 00000100:00080000:2.0:1788204358.529054:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204358.529056:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204358.529063:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204358.529070:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064443064448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204358.529076:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204358.529078:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204358.529079:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204358.529081:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3630 00000100:00080000:2.0:1788204358.529083:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204358.529085:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204358.529088:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204358.529096:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064443064576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204358.529105:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443064320:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204358.529107:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204358.529113:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204358.529114:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204358.529120:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3630 00000100:00080000:2.0:1788204358.529123:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204358.529127:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204358.529132:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204358.529145:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443064704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204358.529174:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204358.529208:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204358/real 1788204358] req@ffff90234598ca80 x1875064443064320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204358.529225:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064443064320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204358.529235:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204358.529239:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204358.529253:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443064320:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204358.529264:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443064448:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204358.529284:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204358.529296:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204358/real 1788204358] req@ffff90234598e680 x1875064443064448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204358.529306:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064443064448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204358.529313:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204358.529316:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204358.529319:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443064448:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204358.529340:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443064576:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204358.529357:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204358.529365:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204358/real 1788204358] req@ffff90234598d880 x1875064443064576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204358.529376:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064443064576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204358.529384:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204358.529387:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204358.529390:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443064576:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204358.529395:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443064704:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204358.529408:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204358.529416:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204358/real 1788204358] req@ffff90234598f800 x1875064443064704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204358.529425:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443064704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204358.529430:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204358.529432:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204358.529434:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443064704:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204359.361022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204360.384988:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204361.408999:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204362.433053:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204363.457043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204363.648999:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204363.649009:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204363.649013:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204363.649022:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3635 00000100:00080000:1.0:1788204363.649033:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204363.649040:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204363.649075:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443064832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204363.649101:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204363.649104:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204363.649106:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204363.649111:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3635 00000100:00080000:1.0:1788204363.649114:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204363.649119:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204363.649127:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204363.649136:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443064832:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204363.649143:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443064960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204363.649155:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204363.649159:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204363.649161:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204363.649165:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3635 00000100:00080000:1.0:1788204363.649169:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204363.649172:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204363.649178:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788204363.649185:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204363.649201:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443065088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204363.649212:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204363.649216:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204363.649218:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204363.649223:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3635 00000100:00100000:3.0:1788204363.649224:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204363/real 1788204363] req@ffff90234519ea00 x1875064443064832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204363.649229:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204363.649239:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204363.649245:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204363.649250:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443064832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204363.649261:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204363.649263:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204363.649270:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443065216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204363.649270:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443064832:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204363.649299:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443064960:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204363.649322:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204363.649336:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204363/real 1788204363] req@ffff90234519f100 x1875064443064960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204363.649348:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443064960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204363.649355:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204363.649359:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204363.649362:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443064960:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204363.649368:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443065088:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204363.649387:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204363.649399:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204363/real 1788204363] req@ffff90234519c380 x1875064443065088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204363.649409:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443065088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204363.649417:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204363.649420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204363.649423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443065088:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204363.649428:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443065216:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204363.649446:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204363.649457:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204363/real 1788204363] req@ffff90234519f480 x1875064443065216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204363.649469:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443065216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204363.649476:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204363.649479:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204363.649482:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443065216:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204364.480979:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204365.505049:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204366.529048:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204367.553062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204368.577092:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204368.768993:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204368.769002:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204368.769006:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204368.769016:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3640 00000100:00080000:2.0:1788204368.769026:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204368.769033:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204368.769068:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443065344/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204368.769089:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204368.769091:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204368.769092:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204368.769095:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3640 00000100:00080000:2.0:1788204368.769098:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204368.769101:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204368.769107:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204368.769116:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443065472/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204368.769122:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204368.769124:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204368.769125:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204368.769127:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3640 00000100:00080000:2.0:1788204368.769129:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204368.769131:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204368.769135:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204368.769139:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443065344:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204368.769145:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443065600/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204368.769156:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204368.769158:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204368.769159:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204368.769161:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3640 00000100:00080000:2.0:1788204368.769163:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204368.769165:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204368.769167:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204368.769174:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443065728/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204368.769216:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443065472:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204368.769243:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443065600:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204368.769257:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443065728:192.168.201.130@tcp:8:kworker.0 00010000:00010000:3.0:1788204369.601034:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204370.625067:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204371.649097:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788204371.842035:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:3.0:1788204371.842087:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788204371.842106:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:3.0:1788204371.842141:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:3.0:1788204371.842150:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:3.0:1788204371.842157:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788204371.843047:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204368/real 1788204371] req@ffff90234598ce00 x1875064443065600/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204371.843077:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443065600/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204371.843088:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204371.843092:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204371.843096:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443065600:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204371.843104:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204368/real 1788204371] req@ffff90234598f480 x1875064443065728/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204371.843122:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443065728/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204371.843130:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204371.843133:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204371.843136:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443065728:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204371.843142:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204368/real 1788204371] req@ffff90234598dc00 x1875064443065344/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204371.843154:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443065344/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204371.843161:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204371.843164:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204371.843168:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443065344:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204371.843174:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204368/real 1788204371] req@ffff90234598c700 x1875064443065472/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204371.843184:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443065472/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204371.843191:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204371.843195:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204371.843198:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443065472:192.168.201.130@tcp:38:kworker.0 00010000:00010000:0.0:1788204372.673661:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204373.696984:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204373.889062:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204373.889077:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204373.889081:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204373.889094:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3645 00000100:00080000:1.0:1788204373.889107:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204373.889114:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204373.889153:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064443065856/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204373.889198:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204373.889203:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204373.889205:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204373.889210:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3645 00000100:00080000:1.0:1788204373.889214:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204373.889219:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204373.889229:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204373.889246:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443065984/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204373.889257:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204373.889261:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204373.889263:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204373.889267:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3645 00000100:00080000:1.0:1788204373.889272:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204373.889277:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204373.889282:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204373.889303:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443065856:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204373.889304:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443066112/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204373.889333:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204373.889336:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204373.889338:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204373.889348:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3645 00000100:00080000:1.0:1788204373.889353:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204373.889357:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204373.889363:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204373.889377:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443066240/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204373.889407:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204373.889488:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204373/real 1788204373] req@ffff90234519dc00 x1875064443065856/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204373.889506:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064443065856/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204373.889518:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204373.889522:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204373.889526:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443065856:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204373.889535:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443065984:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204373.889573:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204373.889586:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204373/real 1788204373] req@ffff90234519f480 x1875064443065984/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204373.889598:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443065984/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204373.889606:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204373.889610:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204373.889613:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443065984:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204373.889642:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443066112:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204373.889665:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204373.889676:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204373/real 1788204373] req@ffff90234519c380 x1875064443066112/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204373.889710:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443066112/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204373.889718:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204373.889720:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204373.889723:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443066112:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204373.889729:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443066240:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204373.889748:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204373.889759:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204373/real 1788204373] req@ffff90234519f100 x1875064443066240/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204373.889776:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443066240/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204373.889783:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204373.889786:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204373.889788:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443066240:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788204374.721023:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204375.745011:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204376.769071:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204377.793035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204378.817085:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204379.009037:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204379.009050:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204379.009055:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204379.009076:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3650 00000100:00080000:3.0:1788204379.009089:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204379.009099:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204379.009147:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443066368/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204379.009181:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204379.009186:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204379.009195:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204379.009201:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3650 00000100:00080000:3.0:1788204379.009206:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204379.009213:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204379.009214:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443066368:192.168.201.130@tcp:250:kworker.0 00000008:00100000:3.0:1788204379.009226:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204379.009249:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443066496/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204379.009261:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204379.009265:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204379.009267:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204379.009272:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3650 00000100:00080000:3.0:1788204379.009276:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204379.009281:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204379.009287:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204379.009301:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443066624/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204379.009302:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204379.009314:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204379.009318:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204379.009321:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204379.009326:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3650 00000100:00080000:3.0:1788204379.009329:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204379.009342:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204379.009348:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204379.009359:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204379/real 1788204379] req@ffff902352689c00 x1875064443066368/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204379.009386:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443066368/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204379.009394:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204379.009397:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204379.009400:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443066368:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204379.009427:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443066496:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204379.009432:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443066752/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204379.009455:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204379.009472:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204379/real 1788204379] req@ffff902352688e00 x1875064443066496/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204379.009484:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443066496/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204379.009496:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204379.009498:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204379.009502:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443066496:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788204379.009506:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443066624:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204379.009522:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204379.009532:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204379/real 1788204379] req@ffff90235268a680 x1875064443066624/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204379.009546:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443066624/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204379.009554:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204379.009558:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204379.009561:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443066624:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788204379.009573:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443066752:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204379.009588:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204379.009596:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204379/real 1788204379] req@ffff902352689500 x1875064443066752/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204379.009603:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443066752/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204379.009608:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204379.009611:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204379.009614:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443066752:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204379.841039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204380.865086:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204381.889004:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204382.913042:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204383.937005:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204384.128971:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204384.128980:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204384.128983:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204384.128991:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3655 00000100:00080000:3.0:1788204384.129001:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204384.129008:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204384.129041:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443066880/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204384.129078:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204384.129085:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204384.129086:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204384.129091:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3655 00000100:00080000:3.0:1788204384.129094:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204384.129099:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204384.129108:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204384.129117:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443066880:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204384.129122:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443067008/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204384.129134:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204384.129137:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204384.129139:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204384.129142:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3655 00000100:00080000:3.0:1788204384.129145:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204384.129149:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204384.129153:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204384.129163:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443067136/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204384.129172:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204384.129175:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204384.129176:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788204384.129180:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204384.129182:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3655 00000100:00080000:3.0:1788204384.129187:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204384.129191:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204384.129196:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204384.129206:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443067264/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204384.129229:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204384/real 1788204384] req@ffff902352689180 x1875064443066880/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204384.129249:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443066880/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204384.129256:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204384.129260:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204384.129264:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443066880:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204384.129297:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443067008:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788204384.129321:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204384.129332:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204384/real 1788204384] req@ffff902352688700 x1875064443067008/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204384.129345:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443067008/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204384.129353:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204384.129356:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204384.129360:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443067008:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788204384.129365:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443067136:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204384.129380:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204384.129388:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204384/real 1788204384] req@ffff90235268bb80 x1875064443067136/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204384.129396:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443067136/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204384.129401:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204384.129402:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204384.129404:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443067136:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788204384.129407:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443067264:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204384.129416:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204384.129421:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204384/real 1788204384] req@ffff902352689880 x1875064443067264/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204384.129427:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443067264/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204384.129431:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204384.129433:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204384.129434:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443067264:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204384.961031:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204385.985002:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204387.009375:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204388.033022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204389.057029:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204389.248980:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204389.248987:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204389.248990:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204389.248998:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3660 00000100:00080000:0.0:1788204389.249008:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204389.249014:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204389.249050:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443067392/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204389.249077:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204389.249083:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204389.249084:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204389.249087:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3660 00000100:00080000:0.0:1788204389.249090:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204389.249092:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204389.249099:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204389.249110:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064443067520/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204389.249117:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204389.249121:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788204389.249121:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443067392:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204389.249122:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204389.249129:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3660 00000100:00080000:0.0:1788204389.249132:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204389.249134:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204389.249137:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204389.249143:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064443067648/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204389.249149:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204389.249151:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204389.249152:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204389.249155:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3660 00000100:00080000:0.0:1788204389.249157:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204389.249160:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204389.249163:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204389.249169:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064443067776/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204389.249198:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204389.249240:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204389/real 1788204389] req@ffff9023450bb480 x1875064443067392/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204389.249260:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443067392/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204389.249271:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204389.249275:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204389.249279:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443067392:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204389.249300:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443067520:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788204389.249321:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204389.249333:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204389/real 1788204389] req@ffff9023450ba680 x1875064443067520/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204389.249345:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064443067520/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204389.249353:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204389.249356:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204389.249359:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443067520:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788204389.249365:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443067648:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204389.249383:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204389.249390:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204389/real 1788204389] req@ffff9023450b8e00 x1875064443067648/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204389.249397:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064443067648/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204389.249402:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204389.249404:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204389.249406:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443067648:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788204389.249409:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443067776:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204389.249419:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204389.249424:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204389/real 1788204389] req@ffff9023450bbb80 x1875064443067776/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204389.249430:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064443067776/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204389.249436:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204389.249438:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204389.249441:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443067776:192.168.201.130@tcp:8:kworker.0 00010000:00010000:3.0:1788204390.080996:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204391.105029:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204392.129021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204393.152985:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204394.176998:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204394.368980:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204394.368987:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204394.368991:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204394.368998:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3665 00000100:00080000:1.0:1788204394.369007:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204394.369011:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204394.369040:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443067904/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204394.369060:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204394.369062:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204394.369063:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204394.369073:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3665 00000100:00080000:1.0:1788204394.369081:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788204394.369088:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443067904:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204394.369088:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204394.369111:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204394.369128:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443068032/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204394.369138:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204394.369141:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204394.369144:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204394.369147:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3665 00000800:00000100:0.0:1788204394.369157:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788204394.369159:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204394.369164:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204394.369170:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204394.369179:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443068160/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204394.369185:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204394.369187:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204394.369188:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204394.369191:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3665 00000100:00080000:1.0:1788204394.369193:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204394.369195:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204394.369197:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204394.369207:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443068288/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204394.369213:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204394/real 1788204394] req@ffff90234519ea00 x1875064443067904/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204394.369252:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443067904/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204394.369260:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204394.369263:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204394.369265:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443067904:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204394.369283:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443068032:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204394.369298:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204394.369306:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204394/real 1788204394] req@ffff90234519c000 x1875064443068032/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204394.369313:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443068032/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204394.369318:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204394.369320:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204394.369323:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443068032:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204394.369329:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443068160:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204394.369349:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204394.369359:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204394/real 1788204394] req@ffff90234519c700 x1875064443068160/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204394.369369:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443068160/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204394.369376:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204394.369380:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204394.369383:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443068160:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204394.369388:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443068288:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204394.369404:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204394.369414:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204394/real 1788204394] req@ffff90234519fb80 x1875064443068288/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204394.369424:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443068288/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204394.369431:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204394.369434:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204394.369437:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443068288:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788204395.201043:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204396.225012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204397.249024:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204398.272982:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204399.297047:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204399.488922:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204399.488930:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204399.488933:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204399.488940:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3671 00000100:00080000:1.0:1788204399.488948:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204399.488953:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204399.488982:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443068416/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204399.489017:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204399.489021:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204399.489023:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204399.489028:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3671 00000100:00080000:1.0:1788204399.489031:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204399.489035:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204399.489044:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204399.489057:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443068544/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204399.489063:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443068416:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204399.489067:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204399.489071:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204399.489073:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204399.489077:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3671 00000100:00080000:1.0:1788204399.489082:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204399.489085:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204399.489090:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204399.489104:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443068672/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204399.489112:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204399.489115:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204399.489117:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204399.489121:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3671 00000100:00080000:1.0:1788204399.489125:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204399.489128:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:0.0:1788204399.489133:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:1.0:1788204399.489134:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204399.489149:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443068800/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204399.489186:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204399/real 1788204399] req@ffff90234519df80 x1875064443068416/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204399.489201:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443068416/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204399.489208:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204399.489211:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204399.489214:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443068416:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204399.489231:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443068544:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204399.489252:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204399.489260:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204399/real 1788204399] req@ffff90234519ed80 x1875064443068544/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204399.489271:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443068544/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204399.489278:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204399.489281:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204399.489284:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443068544:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204399.489290:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443068672:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204399.489308:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204399.489329:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204399/real 1788204399] req@ffff90234519d880 x1875064443068672/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204399.489348:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443068672/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204399.489363:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204399.489367:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204399.489371:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443068672:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204399.489378:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443068800:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204399.489396:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204399.489406:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204399/real 1788204399] req@ffff90234519e300 x1875064443068800/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204399.489417:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443068800/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204399.489424:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204399.489427:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204399.489430:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443068800:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204400.321038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204401.345059:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204402.368995:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204403.393012:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204404.417049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204404.608981:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204404.608987:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204404.608990:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204404.608998:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3676 00000100:00080000:3.0:1788204404.609008:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204404.609015:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204404.609049:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443068928/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204404.609073:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204404.609077:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204404.609079:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204404.609082:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3676 00000100:00080000:3.0:1788204404.609086:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204404.609090:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204404.609099:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204404.609111:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443069056/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204404.609118:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204404.609120:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204404.609122:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204404.609129:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3676 00000100:00080000:3.0:1788204404.609132:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204404.609135:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204404.609139:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204404.609144:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443068928:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204404.609150:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443069184/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204404.609158:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204404.609160:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204404.609161:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204404.609163:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3676 00000100:00080000:3.0:1788204404.609165:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204404.609167:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204404.609170:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204404.609176:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443069312/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204404.609210:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204404.609247:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204404/real 1788204404] req@ffff902352688380 x1875064443068928/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204404.609265:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443068928/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204404.609280:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204404.609284:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204404.609288:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443068928:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204404.609295:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443069056:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204404.609318:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204404.609330:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204404/real 1788204404] req@ffff90235268aa00 x1875064443069056/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204404.609342:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443069056/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204404.609350:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204404.609353:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204404.609356:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443069056:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204404.609378:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443069184:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204404.609399:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204404.609410:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204404/real 1788204404] req@ffff90235268ad80 x1875064443069184/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204404.609421:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443069184/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204404.609427:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204404.609429:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204404.609431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443069184:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204404.609435:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443069312:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204404.609448:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204404.609454:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204404/real 1788204404] req@ffff90235268b480 x1875064443069312/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204404.609461:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443069312/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204404.609465:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204404.609466:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204404.609468:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443069312:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204405.441005:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204406.464986:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204407.489014:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204408.513059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204409.537063:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204409.729039:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204409.729049:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204409.729060:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204409.729070:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3681 00000100:00080000:2.0:1788204409.729080:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204409.729087:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204409.729123:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443069440/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204409.729152:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204409.729155:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204409.729156:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204409.729159:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3681 00000100:00080000:2.0:1788204409.729161:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204409.729164:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204409.729171:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204409.729180:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443069568/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204409.729187:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204409.729189:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204409.729191:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204409.729195:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443069440:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204409.729196:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3681 00000100:00080000:2.0:1788204409.729201:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204409.729207:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204409.729211:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204409.729223:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443069696/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204409.729233:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204409.729237:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204409.729238:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204409.729243:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3681 00000100:00080000:2.0:1788204409.729246:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204409.729250:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204409.729254:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204409.729259:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204409.729264:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443069824/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204409.729291:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204409/real 1788204409] req@ffff90234598c380 x1875064443069440/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204409.729305:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443069440/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204409.729312:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204409.729315:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204409.729317:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443069440:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204409.729335:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443069568:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204409.729355:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204409.729366:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204409/real 1788204409] req@ffff90234598d180 x1875064443069568/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204409.729377:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443069568/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204409.729384:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204409.729387:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204409.729390:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443069568:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204409.729395:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443069696:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204409.729413:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204409.729422:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204409/real 1788204409] req@ffff90234598e300 x1875064443069696/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204409.729432:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443069696/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204409.729439:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204409.729441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204409.729444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443069696:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204409.729448:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443069824:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204409.729462:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204409.729470:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204409/real 1788204409] req@ffff90234598df80 x1875064443069824/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204409.729479:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443069824/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204409.729486:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204409.729488:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204409.729491:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443069824:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204410.561073:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204411.585081:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204412.609038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204413.633072:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204414.657003:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204414.848945:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204414.848955:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204414.848959:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204414.848968:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3686 00000100:00080000:2.0:1788204414.848977:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204414.848982:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204414.849012:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064443069952/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204414.849034:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204414.849037:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204414.849039:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204414.849042:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3686 00000100:00080000:2.0:1788204414.849044:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204414.849046:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204414.849053:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204414.849063:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064443070080/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204414.849074:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204414.849078:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788204414.849078:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443069952:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204414.849081:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204414.849087:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3686 00000100:00080000:2.0:1788204414.849090:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204414.849093:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204414.849096:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204414.849105:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064443070208/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204414.849111:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204414.849114:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204414.849116:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204414.849120:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3686 00000100:00080000:2.0:1788204414.849130:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204414.849137:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:3.0:1788204414.849138:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:2.0:1788204414.849145:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204414.849160:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443070336/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204414.849233:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204414/real 1788204414] req@ffff90234598fb80 x1875064443069952/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204414.849248:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064443069952/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204414.849256:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204414.849259:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204414.849261:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443069952:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204414.849278:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443070080:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204414.849293:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204414.849302:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204414/real 1788204414] req@ffff90234598ed80 x1875064443070080/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204414.849399:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064443070080/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204414.849405:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204414.849408:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204414.849411:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443070080:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204414.849415:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443070208:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204414.849479:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204414.849488:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204414/real 1788204414] req@ffff90234598f100 x1875064443070208/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204414.849496:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064443070208/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204414.849501:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204414.849503:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204414.849505:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443070208:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204414.849508:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443070336:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204414.849525:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204414.849535:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204414/real 1788204414] req@ffff90234598df80 x1875064443070336/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204414.849546:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443070336/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204414.849554:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204414.849557:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204414.849560:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443070336:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788204415.681047:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204416.705027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204417.729029:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204418.753024:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204419.777016:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204419.969015:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204419.969024:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204419.969029:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204419.969038:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3691 00000100:00080000:2.0:1788204419.969048:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204419.969055:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204419.969091:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443070464/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204419.969123:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204419.969127:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204419.969128:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204419.969132:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3691 00000100:00080000:2.0:1788204419.969135:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204419.969139:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204419.969148:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204419.969161:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443070464:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204419.969163:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443070592/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204419.969176:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204419.969179:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204419.969181:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204419.969184:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3691 00000100:00080000:2.0:1788204419.969188:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204419.969192:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204419.969197:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204419.969207:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443070720/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204419.969215:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788204419.969225:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204419.969228:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204419.969229:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204419.969233:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3691 00000100:00080000:2.0:1788204419.969235:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204419.969238:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204419.969242:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204419.969249:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443070848/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204419.969264:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204419/real 1788204419] req@ffff90234598e300 x1875064443070464/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204419.969284:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443070464/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204419.969292:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204419.969295:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204419.969297:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443070464:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204419.969315:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443070592:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204419.969331:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204419.969340:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204419/real 1788204419] req@ffff90234598d180 x1875064443070592/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204419.969346:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443070592/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204419.969351:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204419.969353:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204419.969355:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443070592:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204419.969359:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443070720:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204419.969368:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204419.969373:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204419/real 1788204419] req@ffff90234598c380 x1875064443070720/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204419.969380:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443070720/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204419.969385:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204419.969388:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204419.969390:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443070720:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204419.969395:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443070848:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204419.969411:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204419.969421:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204419/real 1788204419] req@ffff90234598c700 x1875064443070848/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204419.969432:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443070848/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204419.969438:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204419.969442:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204419.969445:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443070848:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204420.801023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204421.825035:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204422.849013:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204423.873070:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204424.897020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204425.089072:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204425.089086:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204425.089091:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204425.089104:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3696 00000100:00080000:0.0:1788204425.089117:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204425.089125:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204425.089185:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064443070976/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204425.089217:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204425.089222:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204425.089223:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204425.089227:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3696 00000100:00080000:0.0:1788204425.089231:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204425.089235:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204425.089246:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204425.089260:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064443071104/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204425.089266:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443070976:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204425.089278:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204425.089282:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204425.089284:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204425.089290:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3696 00000100:00080000:0.0:1788204425.089294:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204425.089297:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204425.089303:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204425.089316:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064443071232/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204425.089326:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204425.089330:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204425.089332:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204425.089338:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3696 00000800:00000100:3.0:1788204425.089339:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788204425.089346:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204425.089351:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204425.089357:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204425.089369:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443071360/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204425.089392:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204425/real 1788204425] req@ffff9023450bbb80 x1875064443070976/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204425.089411:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064443070976/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204425.089420:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204425.089423:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204425.089426:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443070976:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204425.089447:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443071104:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204425.089461:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204425.089469:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204425/real 1788204425] req@ffff9023450b8e00 x1875064443071104/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204425.089477:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064443071104/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204425.089482:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204425.089484:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204425.089486:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443071104:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204425.089489:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443071232:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204425.089499:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204425.089505:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204425/real 1788204425] req@ffff9023450ba680 x1875064443071232/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204425.089512:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064443071232/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204425.089521:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204425.089524:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204425.089527:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443071232:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204425.089533:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443071360:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204425.089557:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204425.089594:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204425/real 1788204425] req@ffff9023450bb480 x1875064443071360/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204425.089602:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443071360/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204425.089607:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204425.089609:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204425.089611:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443071360:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204425.921080:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204426.945076:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204427.969082:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204428.993045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204430.018933:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204430.208993:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204430.209004:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204430.209008:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204430.209019:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3701 00000100:00080000:0.0:1788204430.209031:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204430.209037:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204430.209078:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443071488/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204430.209106:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204430.209109:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204430.209110:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204430.209113:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3701 00000100:00080000:0.0:1788204430.209115:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204430.209117:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204430.209125:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204430.209134:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443071616/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204430.209141:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204430.209144:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204430.209145:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204430.209148:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3701 00000100:00080000:0.0:1788204430.209149:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204430.209151:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204430.209154:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204430.209160:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443071744/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204430.209167:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204430.209169:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204430.209171:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204430.209178:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3701 00000100:00080000:0.0:1788204430.209181:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204430.209182:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443071488:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204430.209190:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204430.209196:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204430.209203:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443071872/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204430.209252:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204430.209294:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204430/real 1788204430] req@ffff9023450b8a80 x1875064443071488/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204430.209309:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443071488/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204430.209317:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204430.209321:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204430.209325:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443071488:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204430.209331:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443071616:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204430.209343:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204430.209357:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204430/real 1788204430] req@ffff9023450bb800 x1875064443071616/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204430.209364:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443071616/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204430.209369:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204430.209371:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204430.209373:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443071616:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204430.209391:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443071744:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204430.209402:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204430.209408:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204430/real 1788204430] req@ffff9023450b9500 x1875064443071744/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204430.209415:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443071744/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204430.209419:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204430.209421:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204430.209423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443071744:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204430.209426:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443071872:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204430.209435:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204430.209440:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204430/real 1788204430] req@ffff9023450b8380 x1875064443071872/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204430.209447:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443071872/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204430.209452:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204430.209453:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204430.209455:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443071872:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204431.041075:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204432.065057:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204433.089099:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204434.113036:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204435.137028:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204435.329056:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204435.329065:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204435.329070:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204435.329080:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3706 00000100:00080000:2.0:1788204435.329090:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204435.329096:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204435.329131:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443072000/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204435.329159:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204435.329163:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204435.329164:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204435.329170:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3706 00000100:00080000:2.0:1788204435.329173:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204435.329178:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204435.329187:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204435.329200:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443072128/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204435.329204:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443072000:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204435.329219:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204435.329222:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204435.329224:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204435.329229:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3706 00000100:00080000:2.0:1788204435.329232:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204435.329235:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204435.329240:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204435.329252:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443072256/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204435.329261:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204435.329265:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204435.329267:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204435.329271:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3706 00000100:00080000:2.0:1788204435.329274:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204435.329278:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204435.329283:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204435.329300:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443072384/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204435.329318:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443072128:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204435.329360:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443072256:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204435.329380:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443072384:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788204436.161229:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204437.185011:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204438.209009:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788204438.401096:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:0.0:1788204438.401129:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788204438.401142:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788204438.401169:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788204438.401175:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:0.0:1788204438.401179:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788204438.401220:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204435/real 1788204438] req@ffff90234598f480 x1875064443072128/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204438.401245:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443072128/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204438.401255:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204438.401260:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204438.401264:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443072128:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204438.401272:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204435/real 1788204438] req@ffff90234598ce00 x1875064443072256/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204438.401284:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443072256/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204438.401292:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204438.401295:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204438.401298:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443072256:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204438.401304:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204435/real 1788204438] req@ffff90234598f800 x1875064443072384/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204438.401316:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443072384/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204438.401324:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204438.401327:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204438.401362:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443072384:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204438.401370:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204435/real 1788204438] req@ffff90234598dc00 x1875064443072000/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204438.401381:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443072000/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204438.401388:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204438.401391:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204438.401394:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443072000:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788204439.233047:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204440.257108:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204440.448982:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204440.448993:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204440.448997:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204440.449007:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3711 00000100:00080000:2.0:1788204440.449016:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204440.449023:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204440.449063:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064443072512/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204440.449088:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204440.449092:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204440.449094:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204440.449098:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3711 00000100:00080000:2.0:1788204440.449103:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204440.449107:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204440.449116:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204440.449129:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064443072640/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204440.449133:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443072512:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204440.449147:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204440.449151:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204440.449152:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204440.449157:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3711 00000100:00080000:2.0:1788204440.449160:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204440.449164:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204440.449169:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204440.449180:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064443072768/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204440.449188:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204440.449192:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204440.449194:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204440.449197:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3711 00000100:00080000:2.0:1788204440.449201:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204440.449205:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204440.449209:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788204440.449215:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204440.449226:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443072896/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204440.449256:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204440/real 1788204440] req@ffff90234598d880 x1875064443072512/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204440.449303:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064443072512/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204440.449314:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204440.449317:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204440.449321:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443072512:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204440.449348:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443072640:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204440.449369:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204440.449381:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204440/real 1788204440] req@ffff90234598e680 x1875064443072640/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204440.449391:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064443072640/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204440.449399:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204440.449402:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204440.449405:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443072640:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204440.449411:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443072768:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204440.449440:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204440.449451:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204440/real 1788204440] req@ffff90234598ca80 x1875064443072768/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204440.449462:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064443072768/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204440.449475:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204440.449478:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204440.449481:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443072768:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204440.449486:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443072896:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204440.449504:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204440.449513:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204440/real 1788204440] req@ffff90234598dc00 x1875064443072896/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204440.449536:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443072896/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204440.449542:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204440.449545:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204440.449548:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443072896:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204441.281143:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204442.305013:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204443.329097:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204444.353065:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204445.377051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204445.568990:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204445.568999:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204445.569003:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204445.569011:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3717 00000100:00080000:0.0:1788204445.569021:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204445.569028:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204445.569063:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443073024/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204445.569101:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204445.569105:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204445.569107:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204445.569111:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3717 00000100:00080000:0.0:1788204445.569114:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204445.569118:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204445.569125:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204445.569139:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443073152/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204445.569150:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204445.569153:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204445.569155:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204445.569159:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3717 00000100:00080000:0.0:1788204445.569162:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204445.569168:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204445.569168:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443073024:192.168.201.130@tcp:250:kworker.0 00000008:00100000:0.0:1788204445.569177:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204445.569194:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443073280/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204445.569205:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204445.569220:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204445.569222:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204445.569225:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3717 00000100:00080000:0.0:1788204445.569229:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204445.569238:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:3.0:1788204445.569239:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:0.0:1788204445.569245:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204445.569257:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443073408/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204445.569283:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204445/real 1788204445] req@ffff9023450b9c00 x1875064443073024/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204445.569300:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443073024/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204445.569310:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204445.569314:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204445.569318:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443073024:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204445.569325:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443073152:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204445.569345:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204445.569354:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204445/real 1788204445] req@ffff9023450b9f80 x1875064443073152/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204445.569365:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443073152/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204445.569373:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204445.569376:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204445.569379:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443073152:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204445.569401:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443073280:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204445.569416:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204445.569423:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204445/real 1788204445] req@ffff9023450b8700 x1875064443073280/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204445.569430:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443073280/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204445.569435:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204445.569437:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204445.569438:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443073280:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204445.569441:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443073408:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204445.569450:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204445.569455:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204445/real 1788204445] req@ffff9023450bad80 x1875064443073408/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204445.569461:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443073408/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204445.569466:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204445.569467:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204445.569469:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443073408:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204446.401020:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204447.424993:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204448.449114:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204449.473109:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204450.497091:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204450.689037:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204450.689063:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204450.689068:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204450.689079:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3722 00000100:00080000:0.0:1788204450.689101:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204450.689110:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204450.689183:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443073536/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204450.689214:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204450.689218:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204450.689220:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204450.689224:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3722 00000100:00080000:0.0:1788204450.689228:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204450.689232:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204450.689243:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204450.689266:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443073664/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204450.689276:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204450.689279:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204450.689282:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204450.689287:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3722 00000100:00080000:0.0:1788204450.689291:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204450.689296:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204450.689304:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443073536:192.168.201.130@tcp:250:kworker.0 00000008:00100000:0.0:1788204450.689306:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204450.689324:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443073792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204450.689334:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204450.689338:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204450.689340:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204450.689344:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3722 00000100:00080000:0.0:1788204450.689350:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204450.689355:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204450.689360:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204450.689372:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443073920/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204450.689396:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204450.689449:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204450/real 1788204450] req@ffff9023450baa00 x1875064443073536/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204450.689470:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443073536/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204450.689482:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204450.689486:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204450.689496:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443073536:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204450.689507:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443073664:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204450.689533:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204450.689545:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204450/real 1788204450] req@ffff9023450b9180 x1875064443073664/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204450.689556:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443073664/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204450.689570:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204450.689574:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204450.689578:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443073664:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204450.689606:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443073792:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204450.689632:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204450.689644:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204450/real 1788204450] req@ffff9023450b8000 x1875064443073792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204450.689657:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443073792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204450.689666:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204450.689670:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204450.689673:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443073792:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204450.689742:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443073920:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204450.689760:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204450.689769:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204450/real 1788204450] req@ffff9023450ba300 x1875064443073920/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204450.689780:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443073920/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204450.689946:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204450.689953:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204450.689961:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443073920:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204451.521305:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204452.545118:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204453.569053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204454.593104:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204455.617110:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204455.808979:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204455.808989:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204455.808993:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204455.809003:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3727 00000100:00080000:2.0:1788204455.809023:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204455.809038:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204455.809086:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443074048/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204455.809116:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204455.809119:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204455.809122:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204455.809126:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3727 00000100:00080000:2.0:1788204455.809129:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204455.809135:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204455.809146:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204455.809160:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443074176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204455.809166:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443074048:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204455.809185:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204455.809190:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204455.809192:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204455.809198:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3727 00000100:00080000:2.0:1788204455.809201:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204455.809206:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204455.809212:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204455.809228:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443074304/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204455.809240:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788204455.809243:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204455.809247:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204455.809249:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204455.809254:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3727 00000100:00080000:2.0:1788204455.809261:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204455.809264:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204455.809269:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204455.809280:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204455/real 1788204455] req@ffff90234598f800 x1875064443074048/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204455.809281:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443074432/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204455.809348:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443074048/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204455.809360:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204455.809364:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204455.809368:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443074048:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204455.809391:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443074176:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204455.809416:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204455.809428:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204455/real 1788204455] req@ffff90234598ce00 x1875064443074176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204455.809437:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443074176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204455.809443:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204455.809445:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204455.809447:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443074176:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204455.809451:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443074304:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204455.809478:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204455.809486:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204455/real 1788204455] req@ffff90234598f480 x1875064443074304/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204455.809497:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443074304/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204455.809503:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204455.809505:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204455.809507:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443074304:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204455.809511:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443074432:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204455.809521:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204455.809528:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204455/real 1788204455] req@ffff90234598c700 x1875064443074432/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204455.809534:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443074432/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204455.809538:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204455.809540:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204455.809542:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443074432:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204456.641052:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204457.665010:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204458.689045:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204459.712990:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204460.737032:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204460.928983:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204460.928991:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204460.928994:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204460.929001:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3732 00000100:00080000:0.0:1788204460.929009:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204460.929015:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204460.929046:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443074560/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204460.929067:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204460.929070:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204460.929071:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204460.929073:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3732 00000100:00080000:0.0:1788204460.929075:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204460.929077:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204460.929084:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204460.929091:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443074688/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204460.929098:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204460.929100:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204460.929101:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204460.929104:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3732 00000100:00080000:0.0:1788204460.929105:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204460.929108:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204460.929116:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204460.929120:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443074560:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204460.929125:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443074816/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204460.929133:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204460.929135:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204460.929136:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204460.929138:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3732 00000100:00080000:0.0:1788204460.929140:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204460.929142:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204460.929144:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204460.929150:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443074944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204460.929186:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204460.929222:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204460/real 1788204460] req@ffff9023450ba300 x1875064443074560/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204460.929236:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443074560/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204460.929243:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204460.929252:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204460.929255:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443074560:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204460.929260:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443074688:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204460.929275:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204460.929283:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204460/real 1788204460] req@ffff9023450b8000 x1875064443074688/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204460.929290:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443074688/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204460.929295:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204460.929297:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204460.929299:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443074688:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204460.929315:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443074816:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204460.929325:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204460.929331:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204460/real 1788204460] req@ffff9023450b9180 x1875064443074816/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204460.929337:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443074816/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204460.929343:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204460.929344:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204460.929346:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443074816:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204460.929350:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443074944:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204460.929358:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204460.929363:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204460/real 1788204460] req@ffff9023450baa00 x1875064443074944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204460.929369:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443074944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204460.929374:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204460.929375:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204460.929377:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443074944:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204461.761003:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204462.785072:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204463.809053:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204464.833025:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204465.857043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204466.049003:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204466.049013:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204466.049018:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204466.049027:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3737 00000100:00080000:1.0:1788204466.049039:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204466.049046:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204466.049095:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064443075072/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204466.049123:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204466.049127:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204466.049129:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204466.049132:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3737 00000100:00080000:1.0:1788204466.049136:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204466.049140:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204466.049153:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204466.049168:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443075072:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204466.049170:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443075200/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204466.049190:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204466.049194:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204466.049195:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204466.049199:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3737 00000100:00080000:1.0:1788204466.049210:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204466.049215:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204466.049220:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788204466.049231:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204466.049233:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443075328/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204466.049244:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204466.049247:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204466.049249:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204466.049253:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3737 00000100:00080000:1.0:1788204466.049257:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204466.049260:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204466.049265:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204466.049274:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443075456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204466.049287:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204466/real 1788204466] req@ffff90234519f800 x1875064443075072/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204466.049311:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064443075072/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204466.049319:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204466.049322:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204466.049325:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443075072:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204466.049343:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443075200:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204466.049365:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204466.049377:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204466/real 1788204466] req@ffff90234519e300 x1875064443075200/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204466.049390:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443075200/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204466.049396:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204466.049398:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204466.049400:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443075200:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204466.049404:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443075328:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204466.049418:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204466.049425:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204466/real 1788204466] req@ffff90234519d880 x1875064443075328/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204466.049432:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443075328/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204466.049436:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204466.049438:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204466.049440:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443075328:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204466.049443:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443075456:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204466.049451:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204466.049457:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204466/real 1788204466] req@ffff90234519ed80 x1875064443075456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204466.049463:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443075456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204466.049471:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204466.049473:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204466.049474:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443075456:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204466.881015:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204467.905044:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204468.929009:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204469.953019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204470.977005:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204471.168984:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204471.168996:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204471.169000:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204471.169009:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3742 00000100:00080000:3.0:1788204471.169020:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204471.169026:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204471.169062:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064443075584/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204471.169098:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204471.169103:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204471.169104:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204471.169109:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3742 00000100:00080000:3.0:1788204471.169113:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204471.169117:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204471.169125:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204471.169132:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443075584:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204471.169141:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443075712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204471.169153:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204471.169157:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204471.169159:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204471.169163:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3742 00000100:00080000:3.0:1788204471.169166:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204471.169170:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204471.169175:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204471.169185:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443075840/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204471.169195:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788204471.169196:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204471.169201:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204471.169203:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204471.169208:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3742 00000100:00080000:3.0:1788204471.169211:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204471.169216:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204471.169220:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204471.169231:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443075968/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204471.169247:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204471/real 1788204471] req@ffff902352688a80 x1875064443075584/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204471.169271:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064443075584/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204471.169278:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204471.169280:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204471.169286:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443075584:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204471.169303:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443075712:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204471.169318:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204471.169325:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204471/real 1788204471] req@ffff90235268b480 x1875064443075712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204471.169333:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443075712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204471.169338:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204471.169340:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204471.169341:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443075712:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204471.169344:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443075840:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204471.169360:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204471.169369:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204471/real 1788204471] req@ffff90235268ad80 x1875064443075840/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204471.169376:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443075840/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204471.169385:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204471.169386:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204471.169388:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443075840:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204471.169391:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443075968:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204471.169400:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204471.169405:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204471/real 1788204471] req@ffff90235268aa00 x1875064443075968/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204471.169412:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443075968/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204471.169416:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204471.169418:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204471.169419:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443075968:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204472.001029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204473.025038:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204474.049082:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204475.073026:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204476.097055:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204476.288968:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204476.288977:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204476.288981:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204476.288994:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3747 00000100:00080000:3.0:1788204476.289005:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204476.289018:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204476.289050:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443076096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204476.289073:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204476.289075:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204476.289076:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204476.289079:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3747 00000100:00080000:3.0:1788204476.289081:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204476.289084:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204476.289090:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204476.289099:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443076224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204476.289105:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204476.289107:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204476.289108:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204476.289111:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3747 00000100:00080000:3.0:1788204476.289114:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788204476.289118:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443076096:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204476.289120:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204476.289128:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204476.289141:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443076352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204476.289152:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204476.289156:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204476.289157:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204476.289160:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3747 00000100:00080000:3.0:1788204476.289169:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204476.289172:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204476.289175:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204476.289177:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204476.289186:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443076480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204476.289219:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204476/real 1788204476] req@ffff902352688380 x1875064443076096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204476.289235:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443076096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204476.289245:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204476.289249:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204476.289253:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443076096:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204476.289260:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443076224:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204476.289281:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204476.289291:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204476/real 1788204476] req@ffff902352689880 x1875064443076224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204476.289309:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443076224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204476.289316:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204476.289320:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204476.289323:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443076224:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204476.289344:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443076352:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204476.289360:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204476.289367:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204476/real 1788204476] req@ffff90235268bb80 x1875064443076352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204476.289374:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443076352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204476.289379:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204476.289381:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204476.289383:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443076352:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204476.289387:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443076480:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204476.289396:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204476.289406:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204476/real 1788204476] req@ffff902352688700 x1875064443076480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204476.289413:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443076480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204476.289417:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204476.289420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204476.289423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443076480:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204477.120998:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204478.145021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204479.169009:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204480.193025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204481.216980:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204481.408996:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204481.409006:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204481.409010:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204481.409027:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3752 00000100:00080000:1.0:1788204481.409041:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204481.409047:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204481.409082:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443076608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204481.409107:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204481.409110:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204481.409112:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204481.409116:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3752 00000100:00080000:1.0:1788204481.409119:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204481.409122:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204481.409131:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204481.409142:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443076736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204481.409159:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204481.409162:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204481.409164:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204481.409167:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443076608:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204481.409170:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3752 00000100:00080000:1.0:1788204481.409177:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204481.409181:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204481.409185:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204481.409196:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443076864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204481.409205:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204481.409209:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204481.409210:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204481.409214:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3752 00000100:00080000:1.0:1788204481.409218:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204481.409221:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204481.409225:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204481.409230:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204481.409238:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443076992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204481.409285:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204481/real 1788204481] req@ffff90234519df80 x1875064443076608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204481.409310:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443076608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204481.409319:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204481.409322:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204481.409326:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443076608:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204481.409350:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443076736:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204481.409370:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204481.409381:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204481/real 1788204481] req@ffff90234519fb80 x1875064443076736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204481.409392:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443076736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204481.409404:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204481.409407:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204481.409410:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443076736:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204481.409415:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443076864:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204481.409431:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204481.409440:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204481/real 1788204481] req@ffff90234519c700 x1875064443076864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204481.409449:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443076864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204481.409456:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204481.409459:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204481.409461:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443076864:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204481.409466:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443076992:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204481.409480:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204481.409489:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204481/real 1788204481] req@ffff90234519c000 x1875064443076992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204481.409499:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443076992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204481.409506:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204481.409509:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204481.409512:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443076992:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204482.241004:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204483.265031:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204484.289029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204485.313040:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204486.337028:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204486.528976:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204486.528987:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204486.528992:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204486.529001:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3758 00000100:00080000:1.0:1788204486.529012:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204486.529020:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204486.529055:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443077120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204486.529096:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204486.529100:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204486.529102:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204486.529108:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3758 00000100:00080000:1.0:1788204486.529112:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204486.529117:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204486.529126:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204486.529130:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443077120:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204486.529145:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443077248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204486.529158:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204486.529162:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204486.529164:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204486.529168:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3758 00000100:00080000:1.0:1788204486.529172:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204486.529176:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204486.529182:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788204486.529182:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204486.529198:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443077376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204486.529209:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204486.529213:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204486/real 1788204486] req@ffff90234519ea00 x1875064443077120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204486.529214:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204486.529217:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204486.529226:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3758 00000100:00080000:1.0:1788204486.529230:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204486.529233:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204486.529237:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204486.529246:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443077504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204486.529276:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443077120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204486.529284:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204486.529287:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204486.529289:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443077120:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204486.529306:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443077248:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204486.529320:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204486.529328:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204486/real 1788204486] req@ffff90234519f100 x1875064443077248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204486.529338:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443077248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204486.529346:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204486.529350:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204486.529354:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443077248:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204486.529447:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443077376:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204486.529463:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204486.529475:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204486/real 1788204486] req@ffff90234519c380 x1875064443077376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204486.529490:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443077376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204486.529549:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204486.529552:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204486.529555:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443077376:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204486.529560:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443077504:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204486.529575:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204486.529582:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204486/real 1788204486] req@ffff90234519f480 x1875064443077504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204486.529589:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443077504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204486.529594:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204486.529596:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204486.529597:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443077504:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204487.361079:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204488.385018:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204489.409037:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204490.432992:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204491.457029:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204491.648979:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204491.648987:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204491.648990:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204491.648998:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3763 00000100:00080000:1.0:1788204491.649007:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204491.649014:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204491.649047:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064443077632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204491.649065:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204491.649068:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204491.649069:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204491.649072:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3763 00000100:00080000:1.0:1788204491.649074:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204491.649076:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204491.649083:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204491.649092:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443077760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204491.649099:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204491.649101:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204491.649102:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204491.649105:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3763 00000100:00080000:1.0:1788204491.649106:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204491.649108:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204491.649111:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204491.649117:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443077888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204491.649121:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443077632:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204491.649127:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204491.649129:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204491.649130:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204491.649133:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3763 00000100:00080000:1.0:1788204491.649135:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204491.649137:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204491.649140:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204491.649148:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443078016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204491.649174:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204491.649208:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204491/real 1788204491] req@ffff90234519dc00 x1875064443077632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204491.649223:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064443077632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204491.649231:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204491.649234:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204491.649237:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443077632:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204491.649242:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443077760:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204491.649255:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204491.649262:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204491/real 1788204491] req@ffff90234519f480 x1875064443077760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204491.649269:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443077760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204491.649274:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204491.649276:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204491.649278:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443077760:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204491.649293:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443077888:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204491.649303:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204491.649308:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204491/real 1788204491] req@ffff90234519c380 x1875064443077888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204491.649315:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443077888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204491.649323:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204491.649325:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204491.649327:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443077888:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204491.649330:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443078016:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204491.649338:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204491.649343:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204491/real 1788204491] req@ffff90234519f100 x1875064443078016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204491.649349:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443078016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204491.649353:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204491.649355:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204491.649356:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443078016:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204492.481006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204493.505015:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204494.529044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204495.553059:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204496.577104:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204496.769014:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204496.769024:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204496.769028:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204496.769038:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3768 00000100:00080000:0.0:1788204496.769052:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204496.769060:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204496.769106:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443078144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204496.769138:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204496.769142:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204496.769144:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204496.769148:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3768 00000100:00080000:0.0:1788204496.769152:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204496.769156:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204496.769167:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204496.769179:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443078272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204496.769190:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204496.769192:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443078144:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204496.769194:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204496.769198:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204496.769204:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3768 00000100:00080000:0.0:1788204496.769207:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204496.769210:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204496.769216:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204496.769234:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443078400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204496.769246:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204496.769249:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204496.769251:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204496.769255:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3768 00000100:00080000:0.0:1788204496.769258:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204496.769263:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204496.769268:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788204496.769277:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204496.769281:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443078528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204496.769341:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204496/real 1788204496] req@ffff9023450bad80 x1875064443078144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204496.769358:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443078144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204496.769366:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204496.769369:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204496.769372:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443078144:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204496.769395:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443078272:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204496.769411:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204496.769420:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204496/real 1788204496] req@ffff9023450b8700 x1875064443078272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204496.769427:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443078272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204496.769433:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204496.769435:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204496.769437:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443078272:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204496.769440:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443078400:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204496.769459:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204496.769466:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204496/real 1788204496] req@ffff9023450b9f80 x1875064443078400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204496.769473:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443078400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204496.769478:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204496.769480:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204496.769482:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443078400:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204496.769485:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443078528:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204496.769493:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204496.769498:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204496/real 1788204496] req@ffff9023450b9c00 x1875064443078528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204496.769506:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443078528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204496.769510:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204496.769511:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204496.769513:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443078528:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204497.601099:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204498.625120:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204499.649478:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204500.673058:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204501.697074:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204501.888974:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204501.888984:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204501.888988:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204501.888996:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3773 00000100:00080000:0.0:1788204501.889004:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204501.889009:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204501.889042:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443078656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204501.889069:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204501.889072:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204501.889074:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204501.889078:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3773 00000100:00080000:0.0:1788204501.889081:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204501.889086:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204501.889095:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204501.889107:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443078656:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204501.889109:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443078784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204501.889121:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204501.889123:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204501.889124:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204501.889127:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3773 00000100:00080000:0.0:1788204501.889130:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204501.889133:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204501.889137:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204501.889145:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443078912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204501.889150:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204501.889152:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204501.889153:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204501.889155:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3773 00000100:00080000:0.0:1788204501.889158:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204501.889161:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204501.889163:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204501.889170:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443079040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204501.889197:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443078784:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204501.889221:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443078912:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204501.889258:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443079040:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204502.721178:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204503.745135:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204504.769037:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788204504.961269:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:1.0:1788204504.961334:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788204504.961355:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788204504.962122:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204501/real 1788204504] req@ffff9023450b8380 x1875064443078656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204526 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204504.962154:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443078656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204526 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204504.962168:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204504.962173:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204504.962188:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443078656:192.168.201.130@tcp:250:kworker.0 00000800:00000100:1.0:1788204504.962305:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204504.962328:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204504.962337:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788204504.962509:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204501/real 1788204504] req@ffff9023450b9500 x1875064443078784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204504.962529:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443078784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204504.962540:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204504.962544:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204504.962548:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443078784:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204504.962557:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204501/real 1788204504] req@ffff9023450bb800 x1875064443078912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204504.962571:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443078912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204504.962581:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204504.962584:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204504.962588:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443078912:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204504.962596:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204501/real 1788204504] req@ffff9023450b8a80 x1875064443079040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204504.962616:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443079040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204504.962624:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204504.962628:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204504.962631:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443079040:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204505.793017:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204506.817053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204507.008992:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204507.009000:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204507.009003:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204507.009011:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3778 00000100:00080000:2.0:1788204507.009019:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204507.009024:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204507.009051:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443079168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204507.009076:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204507.009078:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204507.009080:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204507.009082:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3778 00000100:00080000:2.0:1788204507.009084:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204507.009087:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204507.009095:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204507.009105:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443079296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204507.009113:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204507.009116:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204507.009129:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204507.009133:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3778 00000100:00080000:2.0:1788204507.009136:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204507.009139:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204507.009153:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204507.009164:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443079424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204507.009173:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204507.009176:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204507.009178:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204507.009181:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3778 00000100:00080000:2.0:1788204507.009184:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204507.009187:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204507.009191:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204507.009200:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443079552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204507.009280:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443079168:192.168.201.130@tcp:250:kworker.0 00000800:00000100:0.0:1788204507.009351:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204507.009402:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204507/real 1788204507] req@ffff90234598c380 x1875064443079168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204507.009420:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443079168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204507.009440:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204507.009444:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204507.009448:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443079168:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204507.009455:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443079296:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204507.009479:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204507.009491:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204507/real 1788204507] req@ffff90234598d180 x1875064443079296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204507.009514:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443079296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204507.009522:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204507.009525:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204507.009528:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443079296:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204507.009533:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443079424:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204507.009554:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204507.009565:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204507/real 1788204507] req@ffff90234598e300 x1875064443079424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204507.009575:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443079424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204507.009582:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204507.009585:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204507.009588:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443079424:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204507.009592:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443079552:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204507.009610:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204507.009620:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204507/real 1788204507] req@ffff90234598df80 x1875064443079552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204507.009631:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443079552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204507.009638:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204507.009641:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204507.009644:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443079552:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788204507.841830:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204508.865014:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204509.889032:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204510.913098:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204511.937022:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204512.128987:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204512.128994:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204512.128998:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204512.129005:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3783 00000100:00080000:3.0:1788204512.129013:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204512.129025:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204512.129067:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443079680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204512.129087:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204512.129090:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204512.129092:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204512.129096:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3783 00000100:00080000:3.0:1788204512.129100:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204512.129105:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204512.129113:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204512.129125:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443079808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204512.129128:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443079680:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204512.129134:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204512.129137:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204512.129138:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204512.129142:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3783 00000100:00080000:3.0:1788204512.129145:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204512.129149:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204512.129152:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204512.129163:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443079936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204512.129170:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204512.129173:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204512.129174:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204512.129177:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3783 00000100:00080000:3.0:1788204512.129181:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204512.129184:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204512.129187:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204512.129189:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204512.129195:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443080064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204512.129230:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204512/real 1788204512] req@ffff902352689180 x1875064443079680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204512.129249:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443079680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204512.129259:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204512.129263:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204512.129267:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443079680:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204512.129287:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443079808:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204512.129304:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204512.129320:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204512/real 1788204512] req@ffff902352689500 x1875064443079808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204512.129328:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443079808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204512.129333:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204512.129335:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204512.129337:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443079808:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204512.129340:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443079936:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204512.129352:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204512.129358:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204512/real 1788204512] req@ffff90235268a680 x1875064443079936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204512.129364:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443079936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204512.129369:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204512.129377:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204512.129379:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443079936:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204512.129385:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443080064:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204512.129402:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204512.129412:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204512/real 1788204512] req@ffff902352688e00 x1875064443080064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204512.129422:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443080064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204512.129430:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204512.129433:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204512.129436:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443080064:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204512.961068:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204513.985059:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204515.009061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204516.033024:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204517.057052:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204517.248999:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204517.249007:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204517.249010:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204517.249018:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3788 00000100:00080000:1.0:1788204517.249026:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204517.249031:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204517.249063:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443080192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204517.249085:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204517.249088:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204517.249090:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204517.249094:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3788 00000100:00080000:1.0:1788204517.249097:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204517.249100:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204517.249109:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204517.249123:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443080320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204517.249134:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443080192:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204517.249134:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204517.249139:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204517.249142:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204517.249145:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3788 00000100:00080000:1.0:1788204517.249148:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204517.249150:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204517.249154:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204517.249163:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443080448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204517.249169:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204517.249171:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204517.249172:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204517.249174:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3788 00000100:00080000:1.0:1788204517.249180:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204517.249183:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204517.249186:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204517.249191:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204517.249194:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443080576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204517.249236:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204517/real 1788204517] req@ffff90234519ea00 x1875064443080192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204517.249251:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443080192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204517.249259:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204517.249262:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204517.249265:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443080192:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204517.249281:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443080320:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204517.249295:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204517.249302:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204517/real 1788204517] req@ffff90234519c000 x1875064443080320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204517.249309:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443080320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204517.249313:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204517.249315:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204517.249317:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443080320:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204517.249320:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443080448:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204517.249329:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204517.249334:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204517/real 1788204517] req@ffff90234519c700 x1875064443080448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204517.249341:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443080448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204517.249349:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204517.249350:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204517.249352:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443080448:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204517.249355:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443080576:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204517.249363:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204517.249368:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204517/real 1788204517] req@ffff90234519fb80 x1875064443080576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204517.249374:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443080576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204517.249379:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204517.249382:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204517.249385:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443080576:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204518.081041:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204519.105069:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204520.129011:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204521.153038:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204522.177035:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204522.368999:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204522.369010:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204522.369014:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204522.369024:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3793 00000100:00080000:3.0:1788204522.369035:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204522.369042:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204522.369078:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443080704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204522.369109:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204522.369113:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204522.369116:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204522.369121:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3793 00000100:00080000:3.0:1788204522.369124:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204522.369129:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204522.369139:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204522.369158:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443080704:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204522.369158:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443080832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204522.369178:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204522.369182:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204522.369185:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204522.369190:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3793 00000100:00080000:3.0:1788204522.369194:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204522.369199:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204522.369204:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204522.369218:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443080960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204522.369227:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204522.369234:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204522.369239:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204522.369241:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204522.369247:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3793 00000100:00080000:3.0:1788204522.369250:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204522.369254:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204522.369260:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204522.369268:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204522/real 1788204522] req@ffff902352689c00 x1875064443080704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204522.369273:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443081088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204522.369300:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443080704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204522.369316:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204522.369321:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204522.369324:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443080704:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204522.369345:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443080832:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204522.369370:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204522.369383:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204522/real 1788204522] req@ffff902352688000 x1875064443080832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204522.369394:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443080832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204522.369402:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204522.369405:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204522.369408:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443080832:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204522.369413:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443080960:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204522.369432:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204522.369443:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204522/real 1788204522] req@ffff90235268b800 x1875064443080960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204522.369453:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443080960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204522.369461:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204522.369464:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204522.369467:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443080960:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204522.369473:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443081088:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204522.369493:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204522.369504:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204522/real 1788204522] req@ffff90235268a300 x1875064443081088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204522.369518:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443081088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204522.369525:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204522.369528:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204522.369531:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443081088:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204523.201061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204524.225041:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204525.249048:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204526.273016:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204527.297011:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204527.488961:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204527.488968:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204527.488972:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204527.488979:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3799 00000100:00080000:3.0:1788204527.488987:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204527.488992:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204527.489019:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064443081216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204527.489042:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204527.489045:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204527.489046:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204527.489049:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3799 00000100:00080000:3.0:1788204527.489051:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204527.489055:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204527.489064:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204527.489079:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443081344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204527.489087:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443081216:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204527.489093:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204527.489098:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204527.489100:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204527.489104:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3799 00000100:00080000:3.0:1788204527.489106:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204527.489109:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204527.489113:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204527.489123:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443081472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204527.489139:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204527.489147:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204527.489149:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204527.489154:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3799 00000800:00000100:0.0:1788204527.489158:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204527.489160:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204527.489166:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204527.489172:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204527.489185:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443081600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204527.489207:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204527/real 1788204527] req@ffff90235268b100 x1875064443081216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204527.489227:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064443081216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204527.489237:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204527.489241:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204527.489245:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443081216:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204527.489265:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443081344:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204527.489287:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204527.489300:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204527/real 1788204527] req@ffff90235268a300 x1875064443081344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204527.489312:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443081344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204527.489320:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204527.489323:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204527.489325:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443081344:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204527.489329:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443081472:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204527.489350:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204527.489357:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204527/real 1788204527] req@ffff90235268b800 x1875064443081472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204527.489364:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443081472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204527.489369:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204527.489371:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204527.489372:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443081472:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204527.489375:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443081600:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204527.489385:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204527.489390:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204527/real 1788204527] req@ffff902352688000 x1875064443081600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204527.489400:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443081600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204527.489405:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204527.489411:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204527.489412:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443081600:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204528.321049:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204529.345006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204530.369018:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204531.393053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204532.417083:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204532.608988:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204532.608996:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204532.608999:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204532.609007:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3804 00000100:00080000:0.0:1788204532.609015:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204532.609020:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204532.609050:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443081728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204532.609071:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204532.609073:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204532.609074:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204532.609077:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3804 00000100:00080000:0.0:1788204532.609079:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204532.609081:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204532.609088:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204532.609100:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443081728:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204532.609101:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443081856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204532.609113:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204532.609115:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204532.609116:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204532.609118:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3804 00000100:00080000:0.0:1788204532.609121:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204532.609125:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204532.609129:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204532.609141:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443081984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204532.609150:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204532.609153:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204532.609155:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204532.609158:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3804 00000100:00080000:0.0:1788204532.609162:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788204532.609162:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788204532.609169:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204532.609175:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204532.609186:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443082112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204532.609211:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204532/real 1788204532] req@ffff9023450b8a80 x1875064443081728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204532.609231:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443081728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204532.609238:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204532.609241:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204532.609243:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443081728:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204532.609263:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443081856:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204532.609286:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204532.609298:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204532/real 1788204532] req@ffff9023450bb800 x1875064443081856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204532.609307:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443081856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204532.609313:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204532.609315:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204532.609317:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443081856:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204532.609321:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443081984:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204532.609335:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204532.609351:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204532/real 1788204532] req@ffff9023450b9500 x1875064443081984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204532.609358:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443081984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204532.609363:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204532.609365:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204532.609367:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443081984:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204532.609370:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443082112:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204532.609380:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204532.609386:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204532/real 1788204532] req@ffff9023450bb480 x1875064443082112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204532.609392:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443082112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204532.609396:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204532.609398:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204532.609400:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443082112:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204533.441030:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204534.465015:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204535.489023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204536.513034:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204537.536993:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204537.728988:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204537.728998:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204537.729003:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204537.729012:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3809 00000100:00080000:0.0:1788204537.729023:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204537.729028:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204537.729060:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064443082240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204537.729085:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204537.729088:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204537.729089:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204537.729092:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3809 00000100:00080000:0.0:1788204537.729094:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204537.729096:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204537.729102:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204537.729109:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064443082368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204537.729116:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204537.729118:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204537.729119:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204537.729122:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3809 00000100:00080000:0.0:1788204537.729124:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204537.729128:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204537.729130:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443082240:192.168.201.130@tcp:250:kworker.0 00000008:00100000:0.0:1788204537.729133:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204537.729140:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064443082496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204537.729150:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204537.729152:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204537.729154:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204537.729156:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3809 00000100:00080000:0.0:1788204537.729158:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204537.729160:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204537.729163:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204537.729170:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443082624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204537.729182:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204537.729220:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204537/real 1788204537] req@ffff9023450ba680 x1875064443082240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204537.729233:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064443082240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204537.729240:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204537.729243:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204537.729246:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443082240:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204537.729253:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443082368:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204537.729266:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204537.729275:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204537/real 1788204537] req@ffff9023450b8e00 x1875064443082368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204537.729286:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064443082368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204537.729294:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204537.729297:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204537.729301:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443082368:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204537.729323:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443082496:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204537.729343:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204537.729354:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204537/real 1788204537] req@ffff9023450bbb80 x1875064443082496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204537.729365:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064443082496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204537.729371:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204537.729373:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204537.729375:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443082496:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204537.729379:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443082624:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204537.729391:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204537.729397:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204537/real 1788204537] req@ffff9023450bb480 x1875064443082624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204537.729404:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443082624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204537.729408:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204537.729410:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204537.729411:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443082624:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204538.561062:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204539.585063:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204540.609094:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204541.633017:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204542.657048:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204542.849010:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204542.849020:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204542.849024:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204542.849034:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3814 00000100:00080000:3.0:1788204542.849046:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204542.849052:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204542.849094:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443082752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204542.849119:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204542.849122:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204542.849123:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204542.849126:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3814 00000100:00080000:3.0:1788204542.849128:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204542.849131:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204542.849140:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204542.849149:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443082880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204542.849155:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204542.849159:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788204542.849162:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443082752:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204542.849162:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204542.849167:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3814 00000100:00080000:3.0:1788204542.849172:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204542.849174:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204542.849177:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204542.849184:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443083008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204542.849190:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204542.849193:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204542.849194:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204542.849197:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3814 00000100:00080000:3.0:1788204542.849199:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204542.849202:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204542.849205:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204542.849211:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443083136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204542.849247:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204542.849296:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204542/real 1788204542] req@ffff902352689c00 x1875064443082752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204542.849315:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443082752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204542.849326:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204542.849331:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204542.849335:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443082752:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204542.849359:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443082880:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204542.849384:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204542.849397:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204542/real 1788204542] req@ffff902352688e00 x1875064443082880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204542.849410:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443082880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204542.849418:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204542.849421:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204542.849424:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443082880:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204542.849431:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443083008:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204542.849451:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204542.849462:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204542/real 1788204542] req@ffff90235268a680 x1875064443083008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204542.849474:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443083008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204542.849482:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204542.849485:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204542.849488:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443083008:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204542.849494:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443083136:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204542.849513:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204542.849554:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204542/real 1788204542] req@ffff902352689500 x1875064443083136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204542.849572:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443083136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204542.849579:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204542.849582:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204542.849585:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443083136:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204543.681054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204544.705066:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204545.729084:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204546.753058:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204547.777201:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204547.969040:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204547.969053:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204547.969057:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204547.969070:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3819 00000100:00080000:3.0:1788204547.969084:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204547.969094:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204547.969142:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443083264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204547.969178:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204547.969183:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204547.969184:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204547.969189:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3819 00000100:00080000:3.0:1788204547.969192:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204547.969196:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204547.969207:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204547.969221:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443083392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204547.969230:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443083264:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204547.969234:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204547.969239:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204547.969241:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204547.969246:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3819 00000100:00080000:3.0:1788204547.969250:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204547.969254:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204547.969259:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204547.969272:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443083520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204547.969282:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204547.969285:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204547.969288:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204547.969291:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3819 00000100:00080000:3.0:1788204547.969295:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788204547.969302:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204547.969302:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204547.969309:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204547.969321:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443083648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204547.969362:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204547/real 1788204547] req@ffff902352689180 x1875064443083264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204547.969383:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443083264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204547.969395:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204547.969400:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204547.969404:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443083264:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204547.969443:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443083392:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204547.969462:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204547.969472:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204547/real 1788204547] req@ffff902352688700 x1875064443083392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204547.969484:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443083392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204547.969490:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204547.969492:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204547.969494:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443083392:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204547.969498:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443083520:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204547.969509:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204547.969514:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204547/real 1788204547] req@ffff90235268bb80 x1875064443083520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204547.969529:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443083520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204547.969541:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204547.969544:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204547.969548:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443083520:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204547.969554:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443083648:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204547.969574:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204547.969585:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204547/real 1788204547] req@ffff902352689880 x1875064443083648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204547.969596:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443083648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204547.969609:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204547.969611:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204547.969613:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443083648:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204548.801042:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204549.825058:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204550.849058:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204551.873036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204552.897034:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204553.088940:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204553.088950:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204553.088954:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204553.088963:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3824 00000100:00080000:3.0:1788204553.088973:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204553.088980:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204553.089010:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443083776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204553.089033:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204553.089035:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204553.089037:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204553.089041:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3824 00000100:00080000:3.0:1788204553.089044:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204553.089048:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204553.089057:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204553.089070:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443083904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204553.089074:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443083776:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204553.089088:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204553.089091:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204553.089092:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204553.089096:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3824 00000100:00080000:3.0:1788204553.089098:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204553.089101:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204553.089105:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204553.089113:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443084032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204553.089119:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204553.089121:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204553.089122:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204553.089124:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3824 00000100:00080000:3.0:1788204553.089127:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204553.089129:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204553.089133:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204553.089135:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204553.089141:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443084160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204553.089181:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204553/real 1788204553] req@ffff902352688380 x1875064443083776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204553.089195:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443083776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204553.089202:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204553.089205:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204553.089208:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443083776:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204553.089224:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443083904:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204553.089240:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204553.089247:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204553/real 1788204553] req@ffff90235268aa00 x1875064443083904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204553.089254:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443083904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204553.089259:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204553.089261:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204553.089264:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443083904:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204553.089267:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443084032:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204553.089278:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204553.089283:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204553/real 1788204553] req@ffff90235268ad80 x1875064443084032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204553.089289:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443084032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204553.089294:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204553.089295:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204553.089297:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443084032:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204553.089300:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443084160:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204553.089308:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204553.089313:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204553/real 1788204553] req@ffff90235268b480 x1875064443084160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204553.089320:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443084160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204553.089324:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204553.089325:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204553.089327:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443084160:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204553.921012:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204554.944997:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204555.969028:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204556.992985:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204558.017022:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204558.209036:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204558.209043:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204558.209047:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204558.209057:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3829 00000100:00080000:3.0:1788204558.209068:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204558.209074:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204558.209106:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064443084288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204558.209132:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204558.209136:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204558.209138:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204558.209143:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3829 00000100:00080000:3.0:1788204558.209147:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204558.209151:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204558.209161:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204558.209170:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443084288:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204558.209177:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443084416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204558.209189:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204558.209195:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204558.209196:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204558.209200:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3829 00000100:00080000:3.0:1788204558.209204:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204558.209208:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204558.209213:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204558.209220:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204558.209227:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443084544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204558.209238:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204558.209241:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204558.209242:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204558.209247:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3829 00000100:00100000:0.0:1788204558.209249:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204558/real 1788204558] req@ffff902352688a80 x1875064443084288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204558.209265:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204558.209272:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204558.209280:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204558.209290:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064443084288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204558.209295:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443084672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204558.209300:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204558.209307:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204558.209313:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443084288:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204558.209331:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443084416:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204558.209347:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204558.209357:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204558/real 1788204558] req@ffff90235268b480 x1875064443084416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204558.209366:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443084416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204558.209373:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204558.209375:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204558.209378:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443084416:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204558.209387:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443084544:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204558.209400:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204558.209406:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204558/real 1788204558] req@ffff90235268ad80 x1875064443084544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204558.209419:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443084544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204558.209425:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204558.209427:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204558.209429:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443084544:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204558.209434:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443084672:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204558.209443:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204558.209448:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204558/real 1788204558] req@ffff90235268aa00 x1875064443084672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204558.209455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443084672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204558.209460:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204558.209461:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204558.209463:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443084672:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204559.041030:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204560.065077:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204561.089023:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204562.113022:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204563.137043:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204563.328977:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204563.328984:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204563.328987:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204563.328997:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3834 00000100:00080000:3.0:1788204563.329006:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204563.329013:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204563.329053:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443084800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204563.329076:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204563.329079:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204563.329080:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204563.329083:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3834 00000100:00080000:3.0:1788204563.329086:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204563.329088:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204563.329095:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204563.329104:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443084928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204563.329110:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204563.329113:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204563.329114:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204563.329116:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3834 00000100:00080000:3.0:1788204563.329118:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204563.329122:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204563.329123:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443084800:192.168.201.130@tcp:250:kworker.0 00000008:00100000:3.0:1788204563.329125:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204563.329134:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443085056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204563.329143:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204563.329147:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204563.329149:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204563.329152:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3834 00000100:00080000:3.0:1788204563.329157:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204563.329160:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204563.329176:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204563.329188:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204563.329190:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443085184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204563.329226:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204563/real 1788204563] req@ffff902352688380 x1875064443084800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204563.329242:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443084800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204563.329254:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204563.329257:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204563.329259:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443084800:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204563.329264:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443084928:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204563.329279:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204563.329286:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204563/real 1788204563] req@ffff902352689880 x1875064443084928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204563.329294:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443084928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204563.329298:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204563.329300:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204563.329302:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443084928:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204563.329324:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443085056:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204563.329335:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204563.329341:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204563/real 1788204563] req@ffff90235268bb80 x1875064443085056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204563.329348:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443085056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204563.329356:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204563.329359:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204563.329362:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443085056:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204563.329368:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443085184:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204563.329392:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204563.329404:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204563/real 1788204563] req@ffff902352688700 x1875064443085184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204563.329414:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443085184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204563.329422:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204563.329431:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204563.329434:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443085184:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204564.161007:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204565.185028:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204566.209019:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204567.233074:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204568.257120:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204568.449012:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204568.449022:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204568.449026:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204568.449036:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3839 00000100:00080000:2.0:1788204568.449048:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204568.449054:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204568.449129:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064443085312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204568.449165:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204568.449169:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204568.449171:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204568.449176:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3839 00000100:00080000:2.0:1788204568.449180:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204568.449184:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204568.449196:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204568.449211:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064443085440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204568.449221:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204568.449224:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204568.449227:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204568.449228:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443085312:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204568.449231:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3839 00000100:00080000:2.0:1788204568.449236:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204568.449241:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204568.449245:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204568.449255:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064443085568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204568.449264:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204568.449268:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204568.449269:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204568.449273:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3839 00000100:00080000:2.0:1788204568.449278:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204568.449282:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204568.449288:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204568.449301:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443085696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204568.449369:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443085440:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204568.449404:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443085568:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204568.449427:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443085696:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204569.281020:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204570.304992:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204571.328999:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788204571.521239:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:1.0:1788204571.521296:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788204571.521317:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204571.521378:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204571.521388:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204571.521396:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788204571.521500:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204568/real 1788204571] req@ffff90234598ed80 x1875064443085440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204571.521530:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064443085440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204571.521543:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204571.521550:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204571.521554:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443085440:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204571.521563:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204568/real 1788204571] req@ffff90234598fb80 x1875064443085568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204571.521575:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064443085568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204571.521583:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204571.521587:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204571.521590:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443085568:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204571.521597:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204568/real 1788204571] req@ffff90234598df80 x1875064443085696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204571.521610:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443085696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204571.521618:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204571.521622:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204571.521625:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443085696:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204571.521633:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204568/real 1788204571] req@ffff90234598f100 x1875064443085312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204571.521648:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064443085312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204571.521656:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204571.521661:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204571.521664:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443085312:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788204572.352975:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204573.376976:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204573.568997:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204573.569008:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204573.569012:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204573.569027:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3845 00000100:00080000:0.0:1788204573.569036:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204573.569041:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204573.569071:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443085824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204573.569103:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204573.569107:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204573.569109:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204573.569113:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3845 00000100:00080000:0.0:1788204573.569116:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204573.569121:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204573.569130:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204573.569146:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443085952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204573.569149:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443085824:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204573.569164:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204573.569168:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204573.569169:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204573.569173:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3845 00000100:00080000:0.0:1788204573.569175:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204573.569179:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204573.569183:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204573.569195:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443086080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204573.569203:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204573.569207:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204573.569209:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204573.569212:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3845 00000100:00080000:0.0:1788204573.569216:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204573.569221:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:3.0:1788204573.569222:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:0.0:1788204573.569228:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204573.569241:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443086208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204573.569345:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204573/real 1788204573] req@ffff9023450b9500 x1875064443085824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204573.569367:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443085824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204573.569378:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204573.569382:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204573.569385:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443085824:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204573.569409:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443085952:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204573.569433:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204573.569446:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204573/real 1788204573] req@ffff9023450bb800 x1875064443085952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204573.569458:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443085952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204573.569466:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204573.569469:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204573.569473:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443085952:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204573.569479:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443086080:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204573.569499:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204573.569510:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204573/real 1788204573] req@ffff9023450b8a80 x1875064443086080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204573.569521:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443086080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204573.569530:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204573.569533:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204573.569536:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443086080:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204573.569540:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443086208:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204573.569556:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204573.569566:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204573/real 1788204573] req@ffff9023450b8380 x1875064443086208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204573.569577:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443086208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204573.569584:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204573.569587:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204573.569590:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443086208:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204574.401004:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204575.424996:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204576.449075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204577.473047:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204578.497078:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204578.688997:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204578.689006:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204578.689010:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204578.689019:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3850 00000100:00080000:2.0:1788204578.689029:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204578.689042:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204578.689076:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443086336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204578.689103:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204578.689107:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204578.689109:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204578.689112:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3850 00000100:00080000:2.0:1788204578.689116:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204578.689119:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204578.689128:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204578.689142:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443086464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204578.689153:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443086336:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204578.689163:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204578.689167:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204578.689169:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204578.689174:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3850 00000100:00080000:2.0:1788204578.689177:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204578.689181:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204578.689193:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204578.689204:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443086592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204578.689210:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788204578.689218:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204578.689221:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204578.689224:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204578.689228:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3850 00000100:00080000:2.0:1788204578.689231:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204578.689235:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204578.689239:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204578.689246:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204578/real 1788204578] req@ffff90234598e300 x1875064443086336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204578.689252:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443086720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204578.689262:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443086336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204578.689271:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204578.689275:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204578.689278:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443086336:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204578.689294:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443086464:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204578.689308:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204578.689315:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204578/real 1788204578] req@ffff90234598d180 x1875064443086464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204578.689322:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443086464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204578.689327:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204578.689337:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204578.689339:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443086464:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204578.689343:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443086592:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204578.689354:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204578.689360:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204578/real 1788204578] req@ffff90234598c380 x1875064443086592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204578.689367:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443086592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204578.689376:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204578.689378:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204578.689380:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443086592:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204578.689383:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443086720:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204578.689391:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204578.689396:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204578/real 1788204578] req@ffff90234598c700 x1875064443086720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204578.689402:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443086720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204578.689407:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204578.689408:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204578.689410:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443086720:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204579.521099:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204580.545036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204581.569041:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204582.593109:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204583.617092:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204583.809008:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204583.809018:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204583.809022:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204583.809032:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3855 00000100:00080000:0.0:1788204583.809042:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204583.809048:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204583.809082:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443086848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204583.809119:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204583.809123:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204583.809125:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204583.809129:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3855 00000100:00080000:0.0:1788204583.809133:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204583.809138:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204583.809146:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204583.809157:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443086976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204583.809165:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443086848:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204583.809170:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204583.809175:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204583.809177:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204583.809181:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3855 00000100:00080000:0.0:1788204583.809186:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204583.809190:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204583.809196:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204583.809208:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443087104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204583.809218:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204583.809222:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204583.809227:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788204583.809227:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788204583.809234:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3855 00000100:00080000:0.0:1788204583.809239:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204583.809244:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204583.809249:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204583.809260:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443087232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204583.809282:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204583/real 1788204583] req@ffff9023450b9c00 x1875064443086848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204583.809296:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443086848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204583.809303:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204583.809307:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204583.809311:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443086848:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204583.809335:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443086976:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204583.809357:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204583.809369:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204583/real 1788204583] req@ffff9023450b9f80 x1875064443086976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204583.809381:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443086976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204583.809389:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204583.809392:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204583.809395:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443086976:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204583.809401:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443087104:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204583.809418:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204583.809428:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204583/real 1788204583] req@ffff9023450b8700 x1875064443087104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204583.809438:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443087104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204583.809445:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204583.809448:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204583.809451:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443087104:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204583.809455:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443087232:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204583.809468:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204583.809475:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204583/real 1788204583] req@ffff9023450bad80 x1875064443087232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204583.809486:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443087232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204583.809492:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204583.809495:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204583.809497:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443087232:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204584.641035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204585.665848:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204586.689046:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204587.713022:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204588.737024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204588.928992:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204588.929002:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204588.929012:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204588.929022:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3860 00000100:00080000:1.0:1788204588.929032:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204588.929038:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204588.929072:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443087360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204588.929104:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204588.929107:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204588.929109:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204588.929113:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3860 00000100:00080000:1.0:1788204588.929117:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204588.929121:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204588.929129:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204588.929143:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443087488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204588.929143:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443087360:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204588.929158:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204588.929162:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204588.929164:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204588.929168:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3860 00000100:00080000:1.0:1788204588.929173:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204588.929177:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204588.929181:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204588.929197:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443087616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204588.929200:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788204588.929210:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204588.929214:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204588.929216:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204588.929220:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3860 00000100:00080000:1.0:1788204588.929225:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204588.929228:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204588.929233:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204588.929242:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204588/real 1788204588] req@ffff90234519df80 x1875064443087360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204588.929244:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443087744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204588.929277:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443087360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204588.929287:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204588.929291:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204588.929294:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443087360:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204588.929312:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443087488:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204588.929328:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204588.929336:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204588/real 1788204588] req@ffff90234519ed80 x1875064443087488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204588.929343:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443087488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204588.929362:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204588.929364:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204588.929366:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443087488:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204588.929370:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443087616:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204588.929381:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204588.929387:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204588/real 1788204588] req@ffff90234519d880 x1875064443087616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204588.929393:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443087616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204588.929398:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204588.929400:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204588.929401:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443087616:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204588.929404:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443087744:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204588.929412:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204588.929417:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204588/real 1788204588] req@ffff90234519e300 x1875064443087744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204588.929422:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443087744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204588.929427:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204588.929428:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204588.929430:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443087744:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204589.761008:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204590.785032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204591.809035:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204592.833013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204593.857014:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204594.048956:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204594.048963:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204594.048967:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204594.048975:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3865 00000100:00080000:3.0:1788204594.048984:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204594.048989:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204594.049026:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443087872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204594.049045:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204594.049048:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204594.049050:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204594.049054:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3865 00000100:00080000:3.0:1788204594.049058:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204594.049062:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204594.049070:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204594.049085:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443087872:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204594.049087:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443088000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204594.049103:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204594.049105:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204594.049107:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204594.049110:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3865 00000100:00080000:3.0:1788204594.049114:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204594.049118:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204594.049122:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204594.049134:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443088128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204594.049143:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204594.049147:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1788204594.049151:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204594.049151:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204594.049160:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3865 00000100:00080000:3.0:1788204594.049165:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204594.049169:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204594.049173:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204594.049181:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443088256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204594.049192:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204594/real 1788204594] req@ffff902352689180 x1875064443087872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204594.049211:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443087872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204594.049221:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204594.049225:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204594.049237:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443087872:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204594.049264:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443088000:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204594.049289:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204594.049303:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204594/real 1788204594] req@ffff902352689500 x1875064443088000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204594.049314:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443088000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204594.049322:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204594.049326:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204594.049339:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443088000:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204594.049350:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443088128:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204594.049373:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204594.049388:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204594/real 1788204594] req@ffff90235268a680 x1875064443088128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204594.049400:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443088128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204594.049415:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204594.049419:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204594.049423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443088128:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204594.049429:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443088256:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204594.049450:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204594.049462:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204594/real 1788204594] req@ffff902352688e00 x1875064443088256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204594.049473:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443088256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204594.049480:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204594.049483:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204594.049486:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443088256:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204594.881013:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204595.905034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204596.929036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204597.953000:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204598.977041:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204599.168958:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204599.168965:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204599.168969:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204599.168976:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3870 00000100:00080000:3.0:1788204599.168991:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204599.168998:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204599.169031:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443088384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204599.169051:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204599.169054:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204599.169056:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204599.169058:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3870 00000100:00080000:3.0:1788204599.169061:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204599.169063:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204599.169070:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204599.169079:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443088512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204599.169086:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204599.169089:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788204599.169090:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443088384:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204599.169090:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204599.169102:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3870 00000100:00080000:3.0:1788204599.169105:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204599.169109:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204599.169113:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204599.169139:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443088640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204599.169147:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204599.169150:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204599.169152:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788204599.169154:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204599.169157:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3870 00000100:00080000:3.0:1788204599.169161:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204599.169164:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204599.169168:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204599.169177:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443088768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204599.169196:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204599/real 1788204599] req@ffff902352689c00 x1875064443088384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204599.169213:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443088384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204599.169222:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204599.169226:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204599.169229:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443088384:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204599.169250:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443088512:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204599.169269:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204599.169280:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204599/real 1788204599] req@ffff902352688000 x1875064443088512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204599.169291:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443088512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204599.169298:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204599.169301:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204599.169304:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443088512:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204599.169309:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443088640:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204599.169325:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204599.169348:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204599/real 1788204599] req@ffff90235268b800 x1875064443088640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204599.169359:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443088640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204599.169366:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204599.169369:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204599.169372:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443088640:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204599.169377:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443088768:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204599.169393:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204599.169402:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204599/real 1788204599] req@ffff90235268a300 x1875064443088768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204599.169412:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443088768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204599.169418:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204599.169421:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204599.169424:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443088768:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204600.001012:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204601.025050:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204602.049008:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204603.073003:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204604.097020:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204604.288951:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204604.288960:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204604.288963:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204604.288973:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3875 00000100:00080000:3.0:1788204604.288984:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204604.288991:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204604.289024:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064443088896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204604.289046:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204604.289050:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204604.289051:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204604.289055:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3875 00000100:00080000:3.0:1788204604.289059:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204604.289063:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204604.289072:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204604.289083:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443088896:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204604.289087:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443089024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204604.289097:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204604.289101:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204604.289103:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204604.289106:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3875 00000100:00080000:3.0:1788204604.289109:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204604.289114:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204604.289118:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204604.289128:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443089152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204604.289135:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204604.289143:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204604.289147:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204604.289149:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204604.289153:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3875 00000100:00080000:3.0:1788204604.289157:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204604.289161:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204604.289166:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204604.289167:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204604/real 1788204604] req@ffff90235268b100 x1875064443088896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204604.289178:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443089280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204604.289183:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064443088896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204604.289203:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204604.289208:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204604.289213:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443088896:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204604.289235:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443089024:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204604.289252:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204604.289260:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204604/real 1788204604] req@ffff90235268a300 x1875064443089024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204604.289267:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443089024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204604.289276:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204604.289278:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204604.289280:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443089024:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204604.289283:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443089152:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204604.289292:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204604.289298:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204604/real 1788204604] req@ffff90235268b800 x1875064443089152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204604.289304:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443089152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204604.289309:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204604.289310:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204604.289312:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443089152:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204604.289315:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443089280:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204604.289323:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204604.289328:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204604/real 1788204604] req@ffff902352688000 x1875064443089280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204604.289334:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443089280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204604.289338:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204604.289339:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204604.289342:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443089280:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204605.121014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204606.145087:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204607.169001:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204608.193031:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204609.217014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204609.408991:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204609.409003:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204609.409007:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204609.409016:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3880 00000100:00080000:0.0:1788204609.409028:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204609.409034:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204609.409067:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443089408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204609.409092:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204609.409095:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204609.409097:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204609.409100:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3880 00000100:00080000:0.0:1788204609.409103:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204609.409106:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204609.409115:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204609.409127:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443089536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204609.409131:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443089408:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204609.409146:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204609.409148:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204609.409150:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204609.409155:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3880 00000100:00080000:0.0:1788204609.409159:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204609.409164:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204609.409169:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204609.409182:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443089664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204609.409187:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788204609.409196:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204609.409201:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204609.409202:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204609.409207:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3880 00000100:00080000:0.0:1788204609.409211:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204609.409215:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204609.409219:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204609/real 1788204609] req@ffff9023450baa00 x1875064443089408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788204609.409228:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204609.409237:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443089408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204609.409242:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443089792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204609.409248:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204609.409254:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204609.409259:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443089408:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204609.409278:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443089536:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204609.409293:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204609.409305:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204609/real 1788204609] req@ffff9023450b9180 x1875064443089536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204609.409316:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443089536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204609.409324:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204609.409327:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204609.409330:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443089536:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204609.409336:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443089664:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204609.409372:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204609.409384:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204609/real 1788204609] req@ffff9023450b8000 x1875064443089664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204609.409396:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443089664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204609.409405:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204609.409408:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204609.409411:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443089664:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204609.409416:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443089792:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204609.409430:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204609.409437:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204609/real 1788204609] req@ffff9023450ba300 x1875064443089792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204609.409444:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443089792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204609.409448:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204609.409450:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204609.409452:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443089792:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204610.241007:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204611.265024:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204612.288991:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204613.313009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204614.337044:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204614.529017:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204614.529028:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204614.529038:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204614.529048:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3886 00000100:00080000:3.0:1788204614.529060:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204614.529067:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204614.529109:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443089920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204614.529134:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204614.529136:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204614.529138:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204614.529141:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3886 00000100:00080000:3.0:1788204614.529143:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204614.529146:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204614.529155:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204614.529165:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443090048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204614.529171:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204614.529174:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204614.529175:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204614.529177:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3886 00000100:00080000:3.0:1788204614.529179:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204614.529181:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204614.529184:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204614.529191:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443090176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204614.529199:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204614.529202:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788204614.529203:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443089920:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204614.529204:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204614.529210:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3886 00000100:00080000:3.0:1788204614.529217:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204614.529224:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204614.529228:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204614.529237:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443090304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204614.529298:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204614.529337:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204614/real 1788204614] req@ffff902352689c00 x1875064443089920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204614.529353:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443089920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204614.529364:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204614.529368:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204614.529371:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443089920:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204614.529379:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443090048:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204614.529397:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204614.529408:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204614/real 1788204614] req@ffff902352688e00 x1875064443090048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204614.529419:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443090048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204614.529426:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204614.529429:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204614.529432:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443090048:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204614.529454:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443090176:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204614.529470:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204614.529479:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204614/real 1788204614] req@ffff90235268a680 x1875064443090176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204614.529489:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443090176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204614.529496:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204614.529500:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204614.529503:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443090176:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204614.529509:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443090304:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204614.529539:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204614.529548:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204614/real 1788204614] req@ffff902352689500 x1875064443090304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204614.529557:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443090304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204614.529564:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204614.529567:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204614.529570:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443090304:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204615.361079:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204616.385036:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204617.409022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204618.433028:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204619.457017:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204619.648983:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204619.648991:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204619.648994:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204619.649006:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3891 00000100:00080000:0.0:1788204619.649014:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204619.649019:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204619.649048:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443090432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204619.649068:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204619.649070:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204619.649071:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204619.649074:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3891 00000100:00080000:0.0:1788204619.649076:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204619.649078:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204619.649090:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204619.649103:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443090560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204619.649113:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788204619.649117:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443090432:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204619.649118:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204619.649122:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204619.649129:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3891 00000100:00080000:0.0:1788204619.649132:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204619.649136:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204619.649141:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204619.649154:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443090688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204619.649163:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204619.649167:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204619.649169:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204619.649172:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3891 00000100:00080000:0.0:1788204619.649175:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204619.649181:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:2.0:1788204619.649182:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:0.0:1788204619.649201:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204619.649213:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443090816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204619.649239:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204619/real 1788204619] req@ffff9023450ba300 x1875064443090432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204619.649256:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443090432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204619.649264:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204619.649268:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204619.649271:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443090432:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204619.649301:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443090560:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204619.649322:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204619.649334:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204619/real 1788204619] req@ffff9023450b8000 x1875064443090560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204619.649346:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443090560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204619.649354:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204619.649357:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204619.649359:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443090560:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204619.649363:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443090688:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204619.649377:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204619.649384:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204619/real 1788204619] req@ffff9023450b9180 x1875064443090688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204619.649391:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443090688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204619.649396:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204619.649398:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204619.649400:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443090688:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204619.649403:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443090816:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204619.649411:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204619.649416:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204619/real 1788204619] req@ffff9023450baa00 x1875064443090816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204619.649426:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443090816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204619.649439:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204619.649441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204619.649444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443090816:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204620.481013:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204621.504997:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204622.529024:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204623.553075:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204624.577022:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204624.768963:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204624.768970:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204624.768974:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204624.768981:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3896 00000100:00080000:0.0:1788204624.768990:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204624.768994:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204624.769024:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443090944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204624.769045:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204624.769048:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204624.769049:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204624.769052:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3896 00000100:00080000:0.0:1788204624.769054:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204624.769056:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204624.769062:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204624.769070:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443091072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204624.769077:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204624.769079:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204624.769080:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204624.769082:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3896 00000100:00080000:0.0:1788204624.769084:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204624.769086:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204624.769089:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204624.769093:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443090944:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204624.769109:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443091200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204624.769118:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204624.769125:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204624.769126:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204624.769130:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3896 00000100:00080000:0.0:1788204624.769133:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204624.769136:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204624.769141:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204624.769152:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443091328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204624.769176:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204624.769214:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204624/real 1788204624] req@ffff9023450bad80 x1875064443090944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204624.769231:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443090944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204624.769240:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204624.769242:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204624.769245:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443090944:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204624.769250:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443091072:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204624.769265:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204624.769273:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204624/real 1788204624] req@ffff9023450b8700 x1875064443091072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204624.769283:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443091072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204624.769290:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204624.769293:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204624.769295:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443091072:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204624.769318:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443091200:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204624.769342:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204624.769351:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204624/real 1788204624] req@ffff9023450b9f80 x1875064443091200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204624.769363:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443091200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204624.769370:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204624.769373:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204624.769376:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443091200:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204624.769381:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443091328:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204624.769398:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204624.769409:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204624/real 1788204624] req@ffff9023450b9c00 x1875064443091328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204624.769420:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443091328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204624.769427:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204624.769430:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204624.769433:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443091328:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204625.601018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204626.625032:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204627.649005:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204628.673015:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204629.697029:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204629.888978:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204629.888985:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204629.888989:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204629.888997:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3901 00000100:00080000:3.0:1788204629.889006:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204629.889039:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204629.889071:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443091456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204629.889098:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204629.889101:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204629.889103:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204629.889107:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3901 00000100:00080000:3.0:1788204629.889112:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204629.889116:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204629.889125:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204629.889140:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443091584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204629.889141:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443091456:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204629.889158:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204629.889161:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204629.889164:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204629.889168:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3901 00000100:00080000:3.0:1788204629.889171:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204629.889176:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204629.889181:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204629.889194:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443091712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204629.889201:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204629.889204:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204629.889205:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788204629.889211:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204629.889264:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204629/real 1788204629] req@ffff902352689180 x1875064443091456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204629.889265:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3901 00000100:00080000:3.0:1788204629.889271:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204629.889279:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204629.889286:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204629.889292:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443091456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204629.889298:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443091840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204629.889313:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204629.889319:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204629.889323:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443091456:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204629.889346:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443091584:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204629.889371:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204629.889386:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204629/real 1788204629] req@ffff902352688700 x1875064443091584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204629.889401:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443091584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204629.889410:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204629.889415:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204629.889420:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443091584:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204629.889428:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443091712:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204629.889452:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204629.889465:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204629/real 1788204629] req@ffff90235268bb80 x1875064443091712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204629.889478:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443091712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204629.889487:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204629.889490:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204629.889493:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443091712:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204629.889507:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443091840:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204629.889525:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204629.889536:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204629/real 1788204629] req@ffff902352689880 x1875064443091840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204629.889548:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443091840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204629.889556:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204629.889559:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204629.889562:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443091840:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204630.721029:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204631.745044:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204632.769046:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204633.793118:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204634.817031:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204635.009004:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204635.009015:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204635.009019:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204635.009028:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3906 00000100:00080000:3.0:1788204635.009037:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204635.009045:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204635.009078:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443091968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204635.009103:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204635.009107:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204635.009109:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204635.009114:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3906 00000100:00080000:3.0:1788204635.009118:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204635.009121:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204635.009130:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204635.009143:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443092096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204635.009150:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443091968:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204635.009156:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204635.009160:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204635.009162:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204635.009166:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3906 00000100:00080000:3.0:1788204635.009170:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204635.009173:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204635.009178:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204635.009190:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443092224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204635.009199:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204635.009203:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204635.009205:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204635.009217:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3906 00000100:00080000:3.0:1788204635.009221:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204635.009224:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204635.009229:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204635.009240:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443092352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204635.009273:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443092096:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204635.009307:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443092224:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204635.009330:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443092352:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204635.841165:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204636.865015:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204637.891997:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788204638.081167:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:1.0:1788204638.081208:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788204638.081227:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204638.081261:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204638.081269:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204638.081277:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788204638.081314:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204635/real 1788204638] req@ffff90235268aa00 x1875064443092096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204638.081338:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443092096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204638.081348:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204638.081353:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204638.081358:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443092096:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204638.081365:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204635/real 1788204638] req@ffff90235268ad80 x1875064443092224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204638.081377:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443092224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204638.081385:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204638.081389:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204638.081392:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443092224:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204638.081400:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204635/real 1788204638] req@ffff90235268b480 x1875064443092352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204638.081409:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443092352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204638.081421:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204638.081426:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204638.081429:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443092352:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204638.081435:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204635/real 1788204638] req@ffff902352688380 x1875064443091968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204638.081445:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443091968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204638.081452:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204638.081455:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204638.081458:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443091968:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788204638.913442:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204639.937062:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204640.128957:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204640.128973:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204640.128976:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204640.128985:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3911 00000100:00080000:3.0:1788204640.128994:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204640.129000:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204640.129028:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064443092480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204640.129051:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204640.129055:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204640.129056:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204640.129060:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3911 00000100:00080000:3.0:1788204640.129064:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204640.129067:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204640.129076:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204640.129091:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443092608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204640.129102:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443092480:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204640.129104:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204640.129111:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204640.129113:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204640.129118:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3911 00000100:00080000:3.0:1788204640.129121:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204640.129126:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204640.129131:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204640.129143:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443092736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204640.129152:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204640.129155:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204640.129157:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204640.129161:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3911 00000100:00080000:3.0:1788204640.129164:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204640.129168:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204640.129172:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204640.129184:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443092864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204640.129193:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204640.129243:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204640/real 1788204640] req@ffff902352688a80 x1875064443092480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204640.129261:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064443092480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204640.129269:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204640.129273:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204640.129276:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443092480:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204640.129294:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443092608:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204640.129311:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204640.129323:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204640/real 1788204640] req@ffff902352688380 x1875064443092608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204640.129335:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443092608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204640.129342:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204640.129345:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204640.129347:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443092608:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204640.129353:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443092736:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204640.129371:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204640.129380:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204640/real 1788204640] req@ffff90235268b480 x1875064443092736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204640.129391:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443092736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204640.129398:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204640.129401:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204640.129404:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443092736:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204640.129409:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443092864:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204640.129424:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204640.129433:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204640/real 1788204640] req@ffff90235268ad80 x1875064443092864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204640.129448:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443092864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204640.129454:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204640.129457:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204640.129460:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443092864:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204640.961019:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204641.985077:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204643.009062:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204644.033018:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204645.057068:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204645.249035:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204645.249050:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204645.249054:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204645.249065:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3916 00000100:00080000:0.0:1788204645.249076:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204645.249083:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204645.249122:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443092992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204645.249152:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204645.249155:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204645.249156:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204645.249159:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3916 00000100:00080000:0.0:1788204645.249161:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204645.249163:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204645.249172:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204645.249181:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443093120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204645.249187:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204645.249190:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204645.249191:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204645.249193:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3916 00000100:00080000:0.0:1788204645.249195:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204645.249197:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204645.249200:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204645.249206:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443093248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204645.249211:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443092992:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204645.249214:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204645.249220:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204645.249222:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204645.249225:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3916 00000100:00080000:0.0:1788204645.249226:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204645.249228:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204645.249230:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204645.249238:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443093376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204645.249286:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204645.249330:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204645/real 1788204645] req@ffff9023450b8380 x1875064443092992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204645.249352:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443092992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204645.249359:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204645.249363:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204645.249366:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443092992:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204645.249371:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443093120:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204645.249388:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204645.249396:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204645/real 1788204645] req@ffff9023450b8a80 x1875064443093120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204645.249403:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443093120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204645.249408:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204645.249410:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204645.249412:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443093120:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204645.249433:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443093248:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204645.249443:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204645.249449:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204645/real 1788204645] req@ffff9023450bb800 x1875064443093248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204645.249456:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443093248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204645.249465:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204645.249467:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204645.249469:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443093248:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204645.249472:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443093376:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204645.249483:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204645.249492:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204645/real 1788204645] req@ffff9023450b9500 x1875064443093376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204645.249503:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443093376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204645.249517:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204645.249522:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204645.249524:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443093376:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204646.081063:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204647.105094:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204648.129049:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204649.153094:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204650.177050:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204650.369015:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204650.369027:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204650.369033:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204650.369045:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3921 00000100:00080000:3.0:1788204650.369058:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204650.369066:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204650.369115:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443093504/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204650.369147:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204650.369151:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204650.369154:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204650.369159:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3921 00000100:00080000:3.0:1788204650.369163:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204650.369168:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204650.369179:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204650.369191:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443093504:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204650.369196:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443093632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204650.369208:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204650.369211:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204650.369213:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204650.369217:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3921 00000100:00080000:3.0:1788204650.369220:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204650.369225:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204650.369231:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204650.369245:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443093760/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204650.369255:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204650.369258:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204650.369259:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788204650.369265:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204650.369266:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3921 00000100:00080000:3.0:1788204650.369272:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204650.369277:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204650.369282:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204650.369294:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443093888/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204650.369318:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204650/real 1788204650] req@ffff90235268aa00 x1875064443093504/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204650.369335:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443093504/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204650.369343:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204650.369347:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204650.369351:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443093504:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204650.369379:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443093632:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204650.369404:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204650.369416:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204650/real 1788204650] req@ffff902352689880 x1875064443093632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204650.369428:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443093632/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204650.369437:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204650.369440:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204650.369444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443093632:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204650.369450:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443093760:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204650.369467:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204650.369474:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204650/real 1788204650] req@ffff90235268bb80 x1875064443093760/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204650.369482:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443093760/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204650.369487:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204650.369489:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204650.369491:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443093760:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204650.369493:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443093888:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204650.369503:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204650.369508:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204650/real 1788204650] req@ffff902352688700 x1875064443093888/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204650.369527:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443093888/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204650.369532:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204650.369533:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204650.369535:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443093888:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204651.201042:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204652.225080:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204653.249094:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204654.273042:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204655.297035:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204655.489003:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204655.489011:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204655.489014:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204655.489022:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3927 00000100:00080000:0.0:1788204655.489032:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204655.489038:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204655.489080:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443094016/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204655.489101:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204655.489104:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204655.489105:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204655.489108:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3927 00000100:00080000:0.0:1788204655.489110:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204655.489112:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204655.489125:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204655.489140:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064443094144/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204655.489152:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788204655.489156:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443094016:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204655.489158:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204655.489164:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204655.489170:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3927 00000100:00080000:0.0:1788204655.489174:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204655.489179:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204655.489186:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204655.489198:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064443094272/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204655.489205:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204655.489208:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204655.489209:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788204655.489213:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788204655.489216:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3927 00000100:00080000:0.0:1788204655.489220:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204655.489223:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204655.489227:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204655.489235:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064443094400/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204655.489251:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204655/real 1788204655] req@ffff9023450bb480 x1875064443094016/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204655.489266:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443094016/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204655.489274:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204655.489277:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204655.489280:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443094016:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204655.489298:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443094144:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204655.489313:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204655.489320:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204655/real 1788204655] req@ffff9023450bbb80 x1875064443094144/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204655.489328:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064443094144/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204655.489338:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204655.489341:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204655.489343:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443094144:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204655.489346:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443094272:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204655.489357:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204655.489362:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204655/real 1788204655] req@ffff9023450b8e00 x1875064443094272/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204655.489369:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064443094272/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204655.489375:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204655.489377:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204655.489379:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443094272:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204655.489382:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443094400:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204655.489391:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204655.489396:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204655/real 1788204655] req@ffff9023450ba680 x1875064443094400/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204655.489403:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064443094400/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204655.489408:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204655.489410:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204655.489412:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443094400:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204656.321076:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204657.345107:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204658.369509:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204659.393103:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204660.417045:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204660.609043:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204660.609054:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204660.609059:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204660.609070:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3932 00000100:00080000:3.0:1788204660.609083:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204660.609091:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204660.609134:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443094528/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204660.609160:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204660.609164:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204660.609165:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204660.609169:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3932 00000100:00080000:3.0:1788204660.609171:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204660.609174:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204660.609183:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204660.609193:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443094656/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204660.609202:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788204660.609207:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443094528:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204660.609207:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204660.609210:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204660.609217:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3932 00000100:00080000:3.0:1788204660.609220:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204660.609224:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204660.609227:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204660.609237:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443094784/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204660.609244:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204660.609247:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204660.609249:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204660.609251:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3932 00000100:00080000:3.0:1788204660.609272:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204660.609274:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204660.609277:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788204660.609283:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204660.609288:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443094912/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204660.609330:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204660/real 1788204660] req@ffff902352689180 x1875064443094528/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204660.609347:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443094528/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204660.609355:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204660.609359:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204660.609362:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443094528:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204660.609385:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443094656:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204660.609401:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204660.609410:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204660/real 1788204660] req@ffff902352689500 x1875064443094656/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204660.609418:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443094656/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204660.609424:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204660.609425:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204660.609428:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443094656:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204660.609431:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443094784:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204660.609443:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204660.609450:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204660/real 1788204660] req@ffff90235268a680 x1875064443094784/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204660.609459:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443094784/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204660.609468:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204660.609471:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204660.609475:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443094784:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204660.609492:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443094912:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204660.609515:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204660.609527:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204660/real 1788204660] req@ffff902352688e00 x1875064443094912/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204660.609536:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443094912/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204660.609542:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204660.609544:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204660.609546:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443094912:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204661.441087:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204662.465090:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204663.489036:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204664.513112:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204665.537041:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204665.729059:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204665.729067:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204665.729071:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204665.729081:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3937 00000100:00080000:3.0:1788204665.729092:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204665.729101:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204665.729142:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443095040/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204665.729175:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204665.729179:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204665.729183:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204665.729190:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3937 00000100:00080000:3.0:1788204665.729195:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788204665.729202:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443095040:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204665.729203:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204665.729219:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204665.729239:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443095168/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204665.729254:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204665.729258:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204665.729261:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204665.729267:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3937 00000100:00080000:3.0:1788204665.729272:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204665.729278:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:0.0:1788204665.729284:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:3.0:1788204665.729288:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204665.729304:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443095296/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204665.729314:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204665.729318:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204665.729321:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204665.729325:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3937 00000100:00080000:3.0:1788204665.729329:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788204665.729342:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204665/real 1788204665] req@ffff902352689c00 x1875064443095040/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204665.729361:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443095040/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204665.729371:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204665.729375:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204665.729378:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443095040:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204665.729418:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443095168:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204665.729440:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204665.729450:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204665.729458:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204665/real 1788204665] req@ffff902352688000 x1875064443095168/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788204665.729459:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204665.729479:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443095168/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204665.729489:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204665.729492:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204665.729496:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443095168:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204665.729505:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443095296:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204665.729527:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204665.729540:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204665/real 1788204665] req@ffff90235268b800 x1875064443095296/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204665.729549:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443095296/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204665.729556:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204665.729558:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204665.729561:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443095296:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204665.729604:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443095424/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204665.729656:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443095424:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204665.729679:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204665.729699:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204665/real 1788204665] req@ffff90235268a300 x1875064443095424/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204665.729712:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443095424/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204665.729719:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204665.729723:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204665.729726:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443095424:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204666.561079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204667.585058:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204668.609066:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204669.633063:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204670.657038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204670.848977:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204670.848987:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204670.848991:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204670.849001:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3942 00000100:00080000:3.0:1788204670.849022:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204670.849028:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204670.849062:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064443095552/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204670.849089:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204670.849092:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204670.849094:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204670.849098:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3942 00000100:00080000:3.0:1788204670.849101:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204670.849104:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204670.849112:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204670.849124:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443095680/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204670.849134:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204670.849137:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204670.849139:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204670.849142:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3942 00000100:00080000:3.0:1788204670.849145:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204670.849148:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204670.849153:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204670.849163:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443095808/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204670.849173:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443095552:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204670.849180:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204670.849184:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204670.849186:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204670.849190:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3942 00000100:00080000:3.0:1788204670.849193:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204670.849197:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204670.849203:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204670.849214:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443095936/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204670.849254:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204670.849296:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204670/real 1788204670] req@ffff90235268b100 x1875064443095552/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204670.849326:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064443095552/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204670.849335:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204670.849338:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204670.849341:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443095552:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204670.849347:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443095680:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204670.849365:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204670.849374:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204670/real 1788204670] req@ffff90235268a300 x1875064443095680/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204670.849381:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443095680/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204670.849387:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204670.849389:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204670.849391:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443095680:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204670.849411:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443095808:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204670.849423:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204670.849428:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204670/real 1788204670] req@ffff90235268b800 x1875064443095808/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204670.849436:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443095808/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204670.849441:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204670.849442:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204670.849444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443095808:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204670.849448:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443095936:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204670.849457:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204670.849462:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204670/real 1788204670] req@ffff902352688000 x1875064443095936/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204670.849469:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443095936/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204670.849477:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204670.849479:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204670.849481:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443095936:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204671.681018:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204672.705064:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204673.729033:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204674.753011:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204675.777011:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204675.968988:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204675.968996:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204675.969000:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204675.969009:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3947 00000100:00080000:3.0:1788204675.969019:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204675.969026:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204675.969063:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443096064/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204675.969101:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204675.969105:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204675.969107:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204675.969112:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3947 00000100:00080000:3.0:1788204675.969115:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204675.969119:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204675.969128:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204675.969142:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443096192/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204675.969152:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204675.969156:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204675.969158:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204675.969164:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3947 00000100:00100000:0.0:1788204675.969165:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443096064:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204675.969168:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204675.969175:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204675.969180:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204675.969193:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443096320/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204675.969204:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204675.969208:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204675.969210:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204675.969227:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3947 00000100:00080000:3.0:1788204675.969231:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204675.969234:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204675.969239:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204675.969240:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204675.969256:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443096448/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204675.969300:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204675/real 1788204675] req@ffff902352689c00 x1875064443096064/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204675.969318:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443096064/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204675.969327:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204675.969330:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204675.969333:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443096064:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204675.969340:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443096192:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204675.969374:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204675.969386:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204675/real 1788204675] req@ffff902352688e00 x1875064443096192/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204675.969398:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443096192/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204675.969406:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204675.969409:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204675.969412:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443096192:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204675.969434:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443096320:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204675.969451:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204675.969459:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204675/real 1788204675] req@ffff90235268a680 x1875064443096320/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204675.969484:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443096320/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204675.969492:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204675.969494:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204675.969502:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443096320:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204675.969507:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443096448:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204675.969522:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204675.969530:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204675/real 1788204675] req@ffff902352689500 x1875064443096448/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204675.969539:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443096448/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204675.969546:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204675.969549:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204675.969552:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443096448:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204676.801043:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204677.825035:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204678.849029:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204679.873046:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204680.897056:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204681.089024:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204681.089039:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204681.089044:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204681.089056:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3952 00000100:00080000:1.0:1788204681.089071:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204681.089079:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204681.089144:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064443096576/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204681.089174:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204681.089179:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204681.089180:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204681.089185:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3952 00000100:00080000:1.0:1788204681.089189:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204681.089193:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204681.089203:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204681.089218:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443096704/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204681.089229:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204681.089234:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788204681.089236:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443096576:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204681.089237:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204681.089245:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3952 00000100:00080000:1.0:1788204681.089249:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204681.089252:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204681.089257:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204681.089267:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443096832/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204681.089276:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204681.089280:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204681.089282:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204681.089286:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3952 00000100:00080000:1.0:1788204681.089303:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204681.089307:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204681.089311:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204681.089323:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204681.089324:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443096960/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204681.089382:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204681/real 1788204681] req@ffff90234519f800 x1875064443096576/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204681.089402:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064443096576/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204681.089414:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204681.089417:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204681.089421:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443096576:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204681.089447:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443096704:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204681.089472:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204681.089484:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204681/real 1788204681] req@ffff90234519e300 x1875064443096704/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204681.089495:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443096704/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204681.089503:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204681.089506:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204681.089510:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443096704:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204681.089515:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443096832:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204681.089534:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204681.089545:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204681/real 1788204681] req@ffff90234519d880 x1875064443096832/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204681.089556:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443096832/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204681.089564:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204681.089567:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204681.089584:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443096832:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204681.089591:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443096960:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204681.089610:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204681.089620:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204681/real 1788204681] req@ffff90234519ed80 x1875064443096960/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204681.089632:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443096960/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204681.089640:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204681.089643:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204681.089646:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443096960:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204681.921094:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204682.945069:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204683.969092:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204684.993018:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204686.017069:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204686.209011:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204686.209021:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204686.209025:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204686.209062:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3957 00000100:00080000:1.0:1788204686.209075:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204686.209083:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204686.209126:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443097088/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204686.209155:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204686.209158:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204686.209159:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204686.209162:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3957 00000100:00080000:1.0:1788204686.209165:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204686.209168:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204686.209177:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204686.209185:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443097216/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204686.209192:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204686.209194:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204686.209196:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204686.209199:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3957 00000100:00080000:1.0:1788204686.209200:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204686.209203:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204686.209205:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204686.209211:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443097344/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204686.209218:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788204686.209222:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443097088:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204686.209222:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204686.209223:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204686.209230:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3957 00000100:00080000:1.0:1788204686.209234:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204686.209237:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204686.209239:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204686.209246:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443097472/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204686.209311:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204686.209365:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204686/real 1788204686] req@ffff90234519df80 x1875064443097088/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204686.209384:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443097088/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204686.209392:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204686.209396:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204686.209399:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443097088:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204686.209406:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443097216:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204686.209422:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204686.209430:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204686/real 1788204686] req@ffff90234519fb80 x1875064443097216/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204686.209437:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443097216/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204686.209442:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204686.209444:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204686.209446:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443097216:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204686.209469:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443097344:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204686.209497:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204686.209506:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204686/real 1788204686] req@ffff90234519c700 x1875064443097344/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204686.209513:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443097344/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204686.209522:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204686.209524:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204686.209526:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443097344:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204686.209530:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443097472:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204686.209541:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204686.209546:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204686/real 1788204686] req@ffff90234519c000 x1875064443097472/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204686.209552:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443097472/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204686.209557:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204686.209558:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204686.209560:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443097472:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204687.041111:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204688.065089:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204689.089089:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204690.113066:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204691.137096:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204691.329060:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204691.329076:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204691.329081:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204691.329094:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3962 00000100:00080000:3.0:1788204691.329108:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204691.329116:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204691.329168:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443097600/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204691.329201:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204691.329205:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204691.329207:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204691.329212:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3962 00000100:00080000:3.0:1788204691.329216:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204691.329220:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204691.329231:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204691.329246:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443097728/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204691.329253:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443097600:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204691.329263:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204691.329267:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204691.329269:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204691.329274:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3962 00000100:00080000:3.0:1788204691.329277:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204691.329283:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204691.329288:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204691.329302:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443097856/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204691.329312:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204691.329316:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204691.329318:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204691.329321:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3962 00000100:00080000:3.0:1788204691.329325:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204691.329330:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204691.329335:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204691.329346:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443097984/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204691.329350:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204691.329414:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204691/real 1788204691] req@ffff902352689180 x1875064443097600/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204691.329438:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443097600/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204691.329450:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204691.329457:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204691.329464:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443097600:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204691.329492:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443097728:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204691.329512:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204691.329528:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204691/real 1788204691] req@ffff902352688700 x1875064443097728/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204691.329537:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443097728/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204691.329542:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204691.329544:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204691.329547:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443097728:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204691.329550:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443097856:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204691.329563:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204691.329569:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204691/real 1788204691] req@ffff90235268bb80 x1875064443097856/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204691.329576:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443097856/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204691.329581:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204691.329582:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204691.329584:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443097856:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204691.329587:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443097984:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204691.329597:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204691.329602:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204691/real 1788204691] req@ffff902352689880 x1875064443097984/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204691.329609:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443097984/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204691.329613:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204691.329615:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204691.329617:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443097984:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204692.161094:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204693.185054:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204694.209024:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204695.232994:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204696.257065:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204696.448987:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204696.448997:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204696.449000:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204696.449009:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3967 00000100:00080000:3.0:1788204696.449018:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204696.449024:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204696.449060:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443098112/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204696.449089:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204696.449091:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204696.449093:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204696.449097:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3967 00000100:00080000:3.0:1788204696.449099:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204696.449102:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204696.449109:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204696.449119:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443098240/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204696.449126:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204696.449129:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204696.449130:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204696.449133:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3967 00000100:00080000:3.0:1788204696.449135:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204696.449137:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204696.449140:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204696.449154:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443098368/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204696.449162:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204696.449167:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204696.449169:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204696.449172:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443098112:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204696.449173:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3967 00000100:00080000:3.0:1788204696.449176:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204696.449187:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204696.449193:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204696.449211:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443098496/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204696.449274:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204696.449336:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204696/real 1788204696] req@ffff90235268aa00 x1875064443098112/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204696.449358:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443098112/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204696.449369:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204696.449374:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204696.449378:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443098112:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204696.449387:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443098240:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204696.449410:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204696.449423:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204696/real 1788204696] req@ffff90235268ad80 x1875064443098240/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204696.449435:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443098240/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204696.449444:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204696.449457:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204696.449460:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443098240:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204696.449489:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443098368:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204696.449511:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204696.449523:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204696/real 1788204696] req@ffff90235268b480 x1875064443098368/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204696.449535:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443098368/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204696.449543:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204696.449545:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204696.449547:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443098368:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204696.449552:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443098496:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204696.449577:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204696.449585:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204696/real 1788204696] req@ffff902352688380 x1875064443098496/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204696.449592:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443098496/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204696.449600:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204696.449602:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204696.449604:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443098496:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204697.281085:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204698.305058:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204699.329014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204700.353007:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204701.377059:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204701.568973:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204701.568983:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204701.568988:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204701.568998:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3973 00000100:00080000:3.0:1788204701.569009:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204701.569016:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204701.569052:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064443098624/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204701.569079:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204701.569083:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204701.569085:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204701.569090:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3973 00000100:00080000:3.0:1788204701.569094:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204701.569098:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204701.569107:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204701.569121:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443098752/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204701.569133:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204701.569138:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788204701.569141:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443098624:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204701.569141:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204701.569150:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3973 00000100:00080000:3.0:1788204701.569155:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204701.569159:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204701.569165:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204701.569179:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443098880/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204701.569190:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204701.569193:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204701.569195:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204701.569199:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3973 00000100:00080000:3.0:1788204701.569204:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204701.569208:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204701.569213:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204701.569225:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443099008/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204701.569257:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443098752:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204701.569288:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443098880:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204701.569313:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443099008:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204702.401257:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204703.425076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204704.449025:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788204704.641247:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788204704.641313:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788204704.641335:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788204704.641406:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788204704.641418:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788204704.641427:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788204704.641634:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204701/real 1788204704] req@ffff902352688380 x1875064443098752/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204704.641664:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443098752/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204704.641676:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204704.641682:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204704.641687:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443098752:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204704.641707:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204701/real 1788204704] req@ffff90235268b480 x1875064443098880/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204704.641741:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443098880/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204704.641749:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204704.641754:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204704.641757:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443098880:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204704.641764:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204701/real 1788204704] req@ffff90235268ad80 x1875064443099008/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204704.641777:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443099008/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204704.641784:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204704.641788:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204704.641791:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443099008:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204704.641799:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204701/real 1788204704] req@ffff902352688a80 x1875064443098624/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204704.641812:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064443098624/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204704.641820:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204704.641824:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204704.641827:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443098624:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788204705.473078:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204706.497044:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204706.688980:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204706.688989:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204706.688993:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204706.689015:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3978 00000100:00080000:3.0:1788204706.689025:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204706.689032:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204706.689065:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443099136/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204706.689093:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204706.689097:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204706.689099:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204706.689103:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3978 00000100:00080000:3.0:1788204706.689107:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204706.689111:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204706.689120:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204706.689133:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443099264/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204706.689140:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443099136:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204706.689146:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204706.689150:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204706.689152:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204706.689157:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3978 00000100:00080000:3.0:1788204706.689161:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204706.689165:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204706.689169:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204706.689182:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443099392/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204706.689191:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204706.689194:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204706.689196:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204706.689200:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3978 00000800:00000100:0.0:1788204706.689204:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204706.689206:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204706.689212:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204706.689217:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204706.689229:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443099520/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204706.689244:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204706/real 1788204706] req@ffff90235268aa00 x1875064443099136/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204706.689263:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443099136/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204706.689274:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204706.689277:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204706.689280:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443099136:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204706.689305:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443099264:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204706.689328:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204706.689340:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204706/real 1788204706] req@ffff902352689880 x1875064443099264/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204706.689351:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443099264/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204706.689359:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204706.689363:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204706.689365:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443099264:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204706.689369:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443099392:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204706.689385:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204706.689395:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204706/real 1788204706] req@ffff90235268bb80 x1875064443099392/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204706.689411:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443099392/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204706.689418:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204706.689420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204706.689423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443099392:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204706.689428:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443099520:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204706.689444:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204706.689454:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204706/real 1788204706] req@ffff902352688700 x1875064443099520/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204706.689466:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443099520/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204706.689473:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204706.689476:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204706.689479:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443099520:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204707.521033:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204708.545048:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204709.569039:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204710.593027:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204711.616990:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204711.808989:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204711.808996:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204711.808999:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204711.809006:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3983 00000100:00080000:3.0:1788204711.809015:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204711.809021:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204711.809059:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443099648/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204711.809084:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204711.809087:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204711.809088:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204711.809091:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3983 00000100:00080000:3.0:1788204711.809094:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204711.809096:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204711.809103:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204711.809113:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443099776/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204711.809122:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788204711.809126:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443099648:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204711.809127:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204711.809132:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204711.809138:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3983 00000100:00080000:3.0:1788204711.809142:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204711.809147:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204711.809152:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204711.809166:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443099904/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204711.809173:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204711.809175:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204711.809177:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204711.809179:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3983 00000100:00080000:3.0:1788204711.809182:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788204711.809186:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204711.809186:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204711.809190:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204711.809199:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443100032/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204711.809221:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204711/real 1788204711] req@ffff902352689180 x1875064443099648/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204711.809234:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443099648/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204711.809241:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204711.809243:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204711.809246:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443099648:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204711.809262:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443099776:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204711.809274:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204711.809281:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204711/real 1788204711] req@ffff902352689500 x1875064443099776/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204711.809288:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443099776/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204711.809293:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204711.809295:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204711.809297:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443099776:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204711.809300:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443099904:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204711.809308:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204711.809313:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204711/real 1788204711] req@ffff90235268a680 x1875064443099904/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204711.809319:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443099904/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204711.809324:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204711.809326:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204711.809327:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443099904:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204711.809330:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443100032:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204711.809339:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204711.809343:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204711/real 1788204711] req@ffff902352688e00 x1875064443100032/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204711.809359:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443100032/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204711.809363:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204711.809365:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204711.809367:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443100032:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204712.641006:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204713.665044:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204714.689047:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204715.713019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204716.737027:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204716.929008:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204716.929020:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204716.929025:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204716.929035:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3988 00000100:00080000:0.0:1788204716.929046:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204716.929053:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204716.929095:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064443100160/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204716.929128:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204716.929133:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204716.929135:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204716.929139:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3988 00000100:00080000:0.0:1788204716.929143:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204716.929149:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204716.929159:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204716.929171:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443100160:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204716.929179:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064443100288/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204716.929192:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204716.929196:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204716.929198:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204716.929202:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3988 00000100:00080000:0.0:1788204716.929208:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204716.929212:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204716.929218:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788204716.929227:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204716.929233:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064443100416/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204716.929243:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204716.929246:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204716.929247:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204716.929250:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3988 00000100:00080000:0.0:1788204716.929254:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204716.929257:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204716.929262:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204716.929263:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204716/real 1788204716] req@ffff9023450ba680 x1875064443100160/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204716.929274:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443100544/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204716.929282:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064443100160/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204716.929293:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204716.929297:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204716.929300:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443100160:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204716.929320:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443100288:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204716.929335:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204716.929346:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204716/real 1788204716] req@ffff9023450b8e00 x1875064443100288/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204716.929360:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064443100288/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204716.929369:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204716.929373:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204716.929377:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443100288:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204716.929383:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443100416:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204716.929404:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204716.929416:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204716/real 1788204716] req@ffff9023450bbb80 x1875064443100416/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204716.929429:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064443100416/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204716.929438:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204716.929442:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204716.929445:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443100416:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204716.929451:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443100544:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204716.929468:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204716.929475:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204716/real 1788204716] req@ffff9023450bb480 x1875064443100544/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204716.929482:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443100544/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204716.929488:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204716.929490:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204716.929492:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443100544:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204717.761045:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204718.785047:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204719.809059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204720.833057:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204721.857022:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204722.048994:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204722.049005:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204722.049009:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204722.049016:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3993 00000100:00080000:3.0:1788204722.049025:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204722.049031:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204722.049059:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443100672/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204722.049080:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204722.049083:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204722.049085:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204722.049089:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3993 00000100:00080000:3.0:1788204722.049091:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204722.049094:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204722.049101:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204722.049110:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443100800/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204722.049117:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788204722.049121:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443100672:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204722.049122:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204722.049125:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204722.049131:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3993 00000100:00080000:3.0:1788204722.049133:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204722.049135:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204722.049138:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204722.049146:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443100928/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204722.049152:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204722.049155:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204722.049156:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204722.049158:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3993 00000100:00080000:3.0:1788204722.049167:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204722.049169:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204722.049172:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788204722.049185:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204722.049185:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443101056/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204722.049226:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204722/real 1788204722] req@ffff902352689c00 x1875064443100672/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204722.049244:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443100672/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204722.049255:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204722.049257:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204722.049260:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443100672:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204722.049276:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443100800:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204722.049292:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204722.049299:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204722/real 1788204722] req@ffff902352688000 x1875064443100800/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204722.049307:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443100800/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204722.049314:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204722.049317:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204722.049320:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443100800:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204722.049326:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443100928:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204722.049344:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204722.049354:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204722/real 1788204722] req@ffff90235268b800 x1875064443100928/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204722.049366:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443100928/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204722.049374:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204722.049377:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204722.049380:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443100928:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204722.049386:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443101056:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204722.049410:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204722.049437:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204722/real 1788204722] req@ffff90235268a300 x1875064443101056/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204722.049445:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443101056/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204722.049450:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204722.049451:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204722.049453:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443101056:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204722.881073:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204723.905028:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204724.929034:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204725.953000:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204726.977058:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204727.168964:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204727.168971:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204727.168974:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204727.168982:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 3998 00000100:00080000:3.0:1788204727.168991:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204727.168996:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204727.169032:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064443101184/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204727.169051:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204727.169054:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204727.169055:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204727.169058:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3998 00000100:00080000:3.0:1788204727.169060:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204727.169062:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204727.169068:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204727.169078:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443101312/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204727.169085:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204727.169087:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204727.169088:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204727.169090:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3998 00000100:00080000:3.0:1788204727.169092:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204727.169097:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204727.169099:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443101184:192.168.201.130@tcp:250:kworker.0 00000008:00100000:3.0:1788204727.169100:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204727.169109:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443101440/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204727.169115:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204727.169117:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204727.169118:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204727.169120:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 3998 00000100:00080000:3.0:1788204727.169121:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204727.169123:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204727.169126:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204727.169132:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443101568/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204727.169162:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204727.169190:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204727/real 1788204727] req@ffff90235268b100 x1875064443101184/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204727.169201:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064443101184/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204727.169209:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204727.169211:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204727.169213:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443101184:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204727.169218:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443101312:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204727.169230:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204727.169237:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204727/real 1788204727] req@ffff90235268a300 x1875064443101312/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204727.169244:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443101312/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204727.169249:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204727.169251:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204727.169253:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443101312:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204727.169275:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443101440:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204727.169293:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204727.169304:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204727/real 1788204727] req@ffff90235268b800 x1875064443101440/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204727.169315:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443101440/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204727.169322:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204727.169337:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204727.169341:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443101440:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204727.169346:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443101568:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204727.169362:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204727.169369:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204727/real 1788204727] req@ffff902352688000 x1875064443101568/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204727.169376:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443101568/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204727.169384:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204727.169386:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204727.169388:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443101568:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204728.001045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204729.025042:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204730.049028:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204731.073057:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204732.097039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204732.288993:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204732.289001:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204732.289005:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204732.289013:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4003 00000100:00080000:1.0:1788204732.289022:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204732.289028:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204732.289058:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443101696/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204732.289075:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204732.289078:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204732.289079:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204732.289082:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4003 00000100:00080000:1.0:1788204732.289084:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204732.289086:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204732.289092:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204732.289099:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443101824/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204732.289105:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204732.289108:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204732.289109:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204732.289112:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4003 00000100:00080000:1.0:1788204732.289114:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204732.289117:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204732.289120:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204732.289123:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443101696:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204732.289131:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443101952/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204732.289147:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204732.289151:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204732.289153:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204732.289157:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4003 00000100:00080000:1.0:1788204732.289162:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204732.289167:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204732.289171:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204732.289185:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443102080/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204732.289185:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204732.289224:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204732/real 1788204732] req@ffff90234519ea00 x1875064443101696/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204732.289265:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443101696/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204732.289274:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204732.289277:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204732.289281:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443101696:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204732.289287:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443101824:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204732.289306:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204732.289316:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204732/real 1788204732] req@ffff90234519f100 x1875064443101824/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204732.289327:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443101824/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204732.289334:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204732.289337:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204732.289340:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443101824:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204732.289362:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443101952:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204732.289382:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204732.289392:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204732/real 1788204732] req@ffff90234519c380 x1875064443101952/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204732.289404:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443101952/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204732.289411:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204732.289414:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204732.289417:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443101952:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204732.289422:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443102080:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204732.289438:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204732.289448:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204732/real 1788204732] req@ffff90234519f480 x1875064443102080/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204732.289457:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443102080/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204732.289465:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204732.289467:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204732.289470:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443102080:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204733.121031:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204734.145011:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204735.169026:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204736.193002:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204737.217032:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204737.409005:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204737.409011:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204737.409015:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204737.409024:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4008 00000100:00080000:3.0:1788204737.409032:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204737.409039:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204737.409070:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443102208/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204737.409100:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204737.409104:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204737.409107:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204737.409113:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4008 00000100:00080000:3.0:1788204737.409117:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204737.409122:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204737.409133:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204737.409162:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443102208:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204737.409168:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443102336/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204737.409183:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204737.409187:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204737.409189:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204737.409195:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4008 00000100:00080000:3.0:1788204737.409199:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204737.409203:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204737.409207:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204737.409218:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443102464/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204737.409227:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204737.409227:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204737.409233:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204737.409235:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204737.409239:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4008 00000100:00080000:3.0:1788204737.409242:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204737.409245:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204737.409248:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204737.409257:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443102592/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204737.409266:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204737/real 1788204737] req@ffff902352689c00 x1875064443102208/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204737.409286:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443102208/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204737.409295:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204737.409300:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204737.409303:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443102208:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204737.409328:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443102336:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788204737.409355:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204737.409370:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204737/real 1788204737] req@ffff902352688e00 x1875064443102336/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204737.409380:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443102336/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204737.409386:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204737.409389:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204737.409392:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443102336:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788204737.409399:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443102464:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204737.409418:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204737.409427:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204737/real 1788204737] req@ffff90235268a680 x1875064443102464/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204737.409435:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443102464/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204737.409447:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204737.409449:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204737.409452:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443102464:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788204737.409457:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443102592:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204737.409471:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204737.409479:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204737/real 1788204737] req@ffff902352689500 x1875064443102592/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204737.409489:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443102592/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204737.409494:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204737.409497:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204737.409500:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443102592:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204738.241035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204739.265010:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204740.289058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204741.313069:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204742.337006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204742.528988:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204742.528997:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204742.529000:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204742.529008:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4014 00000100:00080000:3.0:1788204742.529017:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204742.529022:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204742.529095:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443102720/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204742.529127:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204742.529130:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204742.529131:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204742.529135:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4014 00000100:00080000:3.0:1788204742.529138:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204742.529141:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204742.529148:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204742.529160:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443102848/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204742.529168:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204742.529171:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204742.529172:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788204742.529177:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443102720:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204742.529178:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4014 00000100:00080000:3.0:1788204742.529183:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204742.529187:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204742.529191:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204742.529200:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443102976/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204742.529206:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204742.529209:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204742.529210:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204742.529213:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4014 00000100:00080000:3.0:1788204742.529216:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204742.529219:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204742.529222:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204742.529228:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443103104/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204742.529242:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204742.529295:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204742/real 1788204742] req@ffff902352689180 x1875064443102720/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204742.529312:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443102720/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204742.529324:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204742.529327:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204742.529330:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443102720:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204742.529349:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443102848:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788204742.529367:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204742.529376:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204742/real 1788204742] req@ffff902352688700 x1875064443102848/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204742.529384:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443102848/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204742.529389:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204742.529392:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204742.529394:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443102848:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788204742.529397:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443102976:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204742.529408:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204742.529418:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204742/real 1788204742] req@ffff90235268bb80 x1875064443102976/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204742.529430:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443102976/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204742.529448:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204742.529452:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204742.529456:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443102976:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788204742.529462:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443103104:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204742.529479:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204742.529487:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204742/real 1788204742] req@ffff902352689880 x1875064443103104/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204742.529495:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443103104/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204742.529500:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204742.529502:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204742.529505:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443103104:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204743.361012:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204744.385067:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204745.409133:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204746.433102:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204747.457100:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204747.649093:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204747.649108:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204747.649113:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204747.649127:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4019 00000100:00080000:0.0:1788204747.649143:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204747.649152:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204747.649204:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443103232/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204747.649242:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204747.649247:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204747.649249:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204747.649254:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4019 00000100:00080000:0.0:1788204747.649259:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204747.649264:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204747.649277:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204747.649292:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443103360/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204747.649304:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204747.649308:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204747.649313:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788204747.649313:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443103232:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204747.649324:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4019 00000100:00080000:0.0:1788204747.649329:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204747.649335:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204747.649341:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204747.649356:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443103488/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204747.649368:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204747.649372:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204747.649374:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204747.649378:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4019 00000100:00080000:0.0:1788204747.649385:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204747.649390:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204747.649395:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788204747.649402:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204747.649410:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443103616/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204747.649458:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204747/real 1788204747] req@ffff9023450b9500 x1875064443103232/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204747.649477:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443103232/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204747.649487:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204747.649490:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204747.649494:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443103232:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204747.649516:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443103360:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788204747.649534:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204747.649544:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204747/real 1788204747] req@ffff9023450bb800 x1875064443103360/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204747.649553:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443103360/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204747.649558:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204747.649560:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204747.649569:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443103360:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788204747.649573:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443103488:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204747.649587:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204747.649593:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204747/real 1788204747] req@ffff9023450b8a80 x1875064443103488/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204747.649601:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443103488/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204747.649607:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204747.649609:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204747.649612:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443103488:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788204747.649615:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443103616:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204747.649624:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204747.649630:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204747/real 1788204747] req@ffff9023450b8380 x1875064443103616/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204747.649642:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443103616/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204747.649648:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204747.649649:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204747.649651:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443103616:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204748.481075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204749.505073:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204750.529072:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204751.553111:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204752.577051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204752.768979:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204752.768989:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204752.768993:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204752.769002:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4024 00000100:00080000:0.0:1788204752.769013:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204752.769020:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204752.769053:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443103744/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204752.769080:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204752.769084:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204752.769085:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204752.769088:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4024 00000100:00080000:0.0:1788204752.769091:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204752.769094:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204752.769101:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204752.769111:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443103872/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204752.769116:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443103744:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204752.769124:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204752.769128:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204752.769130:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204752.769134:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4024 00000100:00080000:0.0:1788204752.769137:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204752.769141:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204752.769146:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204752.769159:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443104000/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204752.769166:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204752.769168:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204752.769169:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204752.769171:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4024 00000100:00080000:0.0:1788204752.769175:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788204752.769178:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788204752.769181:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204752.769185:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204752.769192:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443104128/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204752.769222:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204752/real 1788204752] req@ffff9023450b9c00 x1875064443103744/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204752.769241:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443103744/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204752.769251:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204752.769255:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204752.769258:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443103744:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204752.769279:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443103872:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788204752.769301:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204752.769314:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204752/real 1788204752] req@ffff9023450b9f80 x1875064443103872/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204752.769334:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443103872/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204752.769342:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204752.769345:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204752.769348:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443103872:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788204752.769353:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443104000:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204752.769368:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204752.769375:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204752/real 1788204752] req@ffff9023450b8700 x1875064443104000/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204752.769382:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443104000/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204752.769387:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204752.769389:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204752.769391:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443104000:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788204752.769394:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443104128:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204752.769402:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204752.769407:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204752/real 1788204752] req@ffff9023450bad80 x1875064443104128/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204752.769414:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443104128/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204752.769421:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204752.769423:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204752.769425:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443104128:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204753.600985:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204754.625043:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204755.649026:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204756.673049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204757.697023:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204757.888974:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204757.888982:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204757.888985:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204757.888992:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4029 00000100:00080000:0.0:1788204757.889000:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204757.889005:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204757.889037:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443104256/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204757.889057:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204757.889059:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204757.889061:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204757.889063:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4029 00000100:00080000:0.0:1788204757.889065:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204757.889067:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204757.889073:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204757.889081:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443104384/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204757.889098:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788204757.889099:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443104256:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204757.889107:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204757.889108:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204757.889111:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4029 00000100:00080000:0.0:1788204757.889114:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204757.889116:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204757.889120:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204757.889128:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443104512/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204757.889134:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204757.889137:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204757.889138:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204757.889140:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4029 00000100:00080000:0.0:1788204757.889144:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204757.889148:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204757.889151:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788204757.889152:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204757.889168:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443104640/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204757.889195:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204757/real 1788204757] req@ffff9023450baa00 x1875064443104256/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204757.889210:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443104256/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204757.889218:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204757.889221:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204757.889224:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443104256:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204757.889241:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443104384:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788204757.889255:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204757.889262:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204757/real 1788204757] req@ffff9023450b9180 x1875064443104384/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204757.889269:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443104384/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204757.889277:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204757.889279:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204757.889281:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443104384:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788204757.889284:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443104512:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204757.889293:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204757.889298:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204757/real 1788204757] req@ffff9023450b8000 x1875064443104512/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204757.889304:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443104512/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204757.889309:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204757.889311:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204757.889312:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443104512:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788204757.889315:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443104640:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204757.889323:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204757.889328:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204757/real 1788204757] req@ffff9023450ba300 x1875064443104640/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204757.889334:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443104640/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204757.889338:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204757.889339:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204757.889342:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443104640:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204758.721028:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204759.745022:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204760.769056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204761.793091:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204762.817038:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204763.009029:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204763.009044:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204763.009050:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204763.009064:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4034 00000100:00080000:3.0:1788204763.009076:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204763.009083:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204763.009129:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443104768/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204763.009158:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204763.009161:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204763.009163:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204763.009167:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4034 00000100:00080000:3.0:1788204763.009169:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204763.009172:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204763.009182:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204763.009192:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064443104896/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204763.009201:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204763.009205:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788204763.009207:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443104768:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204763.009207:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204763.009215:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4034 00000100:00080000:3.0:1788204763.009218:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204763.009221:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204763.009224:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204763.009232:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443105024/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204763.009239:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204763.009241:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204763.009243:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204763.009246:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4034 00000100:00080000:3.0:1788204763.009248:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204763.009250:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204763.009253:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204763.009269:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443105152/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204763.009293:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204763.009362:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204763/real 1788204763] req@ffff90235268aa00 x1875064443104768/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204763.009383:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443104768/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204763.009395:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204763.009399:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204763.009403:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443104768:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204763.009430:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443104896:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788204763.009461:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204763.009477:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204763/real 1788204763] req@ffff902352688a80 x1875064443104896/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204763.009491:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064443104896/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204763.009500:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204763.009503:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204763.009505:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443104896:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788204763.009510:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443105024:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204763.009529:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204763.009537:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204763/real 1788204763] req@ffff90235268ad80 x1875064443105024/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204763.009545:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443105024/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204763.009551:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204763.009553:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204763.009556:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443105024:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788204763.009559:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443105152:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204763.009569:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204763.009575:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204763/real 1788204763] req@ffff90235268b480 x1875064443105152/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204763.009582:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443105152/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204763.009587:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204763.009589:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204763.009591:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443105152:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204763.841071:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204764.865052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204765.889094:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204766.913056:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204767.937114:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204768.129063:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204768.129075:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204768.129079:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204768.129090:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4039 00000100:00080000:1.0:1788204768.129101:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204768.129107:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204768.129141:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064443105280/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204768.129171:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204768.129174:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204768.129175:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204768.129178:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4039 00000100:00080000:1.0:1788204768.129181:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204768.129184:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204768.129191:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204768.129202:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443105408/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204768.129210:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204768.129215:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788204768.129216:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443105280:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204768.129217:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204768.129225:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4039 00000100:00080000:1.0:1788204768.129228:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204768.129230:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204768.129233:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204768.129243:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443105536/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204768.129251:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204768.129253:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204768.129255:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204768.129258:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4039 00000100:00080000:1.0:1788204768.129260:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204768.129262:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204768.129265:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204768.129273:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443105664/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204768.129323:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443105408:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204768.129350:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443105536:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204768.129364:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443105664:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204768.961266:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204769.984994:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204771.009014:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788204771.201162:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:1.0:1788204771.201198:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788204771.201211:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204771.201240:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204771.201248:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204771.201259:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:2.0:1788204771.201285:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204768/real 1788204771] req@ffff90234519f480 x1875064443105408/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204771.201310:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443105408/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204771.201322:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204771.201327:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204771.201331:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443105408:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204771.201340:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204768/real 1788204771] req@ffff90234519c380 x1875064443105536/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204771.201354:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443105536/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204771.201363:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204771.201367:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204771.201372:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443105536:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204771.201380:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204768/real 1788204771] req@ffff90234519f100 x1875064443105664/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204771.201393:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443105664/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204771.201407:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204771.201410:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204771.201414:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443105664:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204771.201421:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204768/real 1788204771] req@ffff90234519dc00 x1875064443105280/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204771.201432:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064443105280/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204771.201439:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204771.201443:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204771.201445:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443105280:192.168.201.130@tcp:250:kworker.0 00010000:00010000:0.0:1788204772.033110:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204773.057657:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204773.249113:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204773.249125:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204773.249130:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204773.249142:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4044 00000100:00080000:3.0:1788204773.249154:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204773.249165:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204773.249216:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443105792/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204773.249261:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204773.249271:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204773.249273:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204773.249278:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4044 00000100:00080000:3.0:1788204773.249283:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204773.249289:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204773.249301:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204773.249319:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443105920/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204773.249341:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204773.249345:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204773.249351:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204773.249355:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4044 00000100:00080000:3.0:1788204773.249359:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204773.249365:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204773.249370:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204773.249383:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443106048/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204773.249392:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204773.249396:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204773.249397:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204773.249401:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4044 00000100:00080000:3.0:1788204773.249405:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204773.249410:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204773.249415:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204773.249426:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064443106176/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204773.249611:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443105792:192.168.201.130@tcp:250:kworker.0 00000800:00000100:1.0:1788204773.249690:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204773.249739:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204773/real 1788204773] req@ffff902352688380 x1875064443105792/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204773.249770:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443105792/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204773.249783:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204773.249789:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204773.249794:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443105792:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204773.249803:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443105920:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788204773.249829:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204773.249843:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204773/real 1788204773] req@ffff90235268b480 x1875064443105920/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204773.249855:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443105920/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204773.249863:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204773.249866:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204773.249869:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443105920:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788204773.249926:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443106048:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204773.249952:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204773.249965:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204773/real 1788204773] req@ffff90235268ad80 x1875064443106048/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204773.249979:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443106048/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204773.249995:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204773.249999:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204773.250002:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443106048:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788204773.250010:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443106176:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204773.250041:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204773.250062:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204773/real 1788204773] req@ffff902352688a80 x1875064443106176/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204773.250076:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064443106176/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204773.250084:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204773.250089:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204773.250092:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443106176:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204774.081077:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204775.105031:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204776.129149:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204777.153084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204778.177000:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204778.369030:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204778.369042:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204778.369047:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204778.369059:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4049 00000100:00080000:1.0:1788204778.369072:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204778.369087:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204778.369136:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443106304/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204778.369171:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204778.369175:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204778.369177:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204778.369181:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4049 00000100:00080000:1.0:1788204778.369185:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204778.369190:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204778.369201:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204778.369212:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443106304:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204778.369218:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443106432/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204778.369229:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204778.369233:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204778.369234:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204778.369238:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4049 00000100:00080000:1.0:1788204778.369241:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204778.369244:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204778.369248:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204778.369259:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443106560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204778.369265:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204778.369268:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204778.369269:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204778.369271:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4049 00000100:00080000:1.0:1788204778.369274:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204778.369276:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204778.369279:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204778.369286:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443106688/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204778.369295:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204778.369345:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204778/real 1788204778] req@ffff90234519ea00 x1875064443106304/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204778.369362:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443106304/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204778.369370:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204778.369373:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204778.369376:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443106304:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204778.369399:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443106432:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204778.369415:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204778.369423:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204778/real 1788204778] req@ffff90234519c000 x1875064443106432/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204778.369430:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443106432/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204778.369435:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204778.369437:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204778.369439:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443106432:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204778.369457:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443106560:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204778.369471:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204778.369478:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204778/real 1788204778] req@ffff90234519c700 x1875064443106560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204778.369484:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443106560/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204778.369490:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204778.369492:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204778.369494:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443106560:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204778.369496:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443106688:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204778.369505:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204778.369509:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204778/real 1788204778] req@ffff90234519fb80 x1875064443106688/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204778.369516:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443106688/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204778.369521:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204778.369522:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204778.369524:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443106688:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204779.201088:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204780.225022:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204781.249027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204782.272991:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204783.297034:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204783.488990:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204783.489000:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204783.489004:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204783.489013:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4055 00000100:00080000:1.0:1788204783.489024:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204783.489030:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204783.489066:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443106816/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204783.489089:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204783.489093:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204783.489094:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204783.489099:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4055 00000100:00080000:1.0:1788204783.489103:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204783.489107:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204783.489117:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204783.489136:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443106944/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204783.489146:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204783.489149:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204783.489151:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788204783.489152:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443106816:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204783.489155:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4055 00000100:00080000:1.0:1788204783.489159:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204783.489163:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204783.489167:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204783.489176:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443107072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204783.489185:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204783.489188:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204783.489190:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204783.489194:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4055 00000100:00080000:1.0:1788204783.489198:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204783.489201:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204783.489206:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788204783.489217:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204783.489219:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443107200/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204783.489260:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204783/real 1788204783] req@ffff90234519df80 x1875064443106816/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204783.489278:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443106816/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204783.489288:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204783.489292:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204783.489295:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443106816:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204783.489317:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443106944:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204783.489336:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204783.489347:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204783/real 1788204783] req@ffff90234519ed80 x1875064443106944/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204783.489372:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443106944/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204783.489380:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204783.489383:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204783.489386:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443106944:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204783.489392:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443107072:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204783.489410:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204783.489421:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204783/real 1788204783] req@ffff90234519d880 x1875064443107072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204783.489432:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443107072/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204783.489437:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204783.489439:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204783.489441:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443107072:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204783.489445:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443107200:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204783.489461:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204783.489471:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204783/real 1788204783] req@ffff90234519e300 x1875064443107200/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204783.489483:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443107200/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204783.489490:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204783.489493:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204783.489496:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443107200:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788204784.321058:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204785.344990:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204786.369077:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204787.393100:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204788.417068:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204788.608953:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204788.608965:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204788.608969:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204788.608979:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4060 00000100:00080000:0.0:1788204788.608990:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204788.608997:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204788.609042:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443107328/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204788.609066:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204788.609069:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204788.609070:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204788.609073:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4060 00000100:00080000:0.0:1788204788.609075:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204788.609078:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204788.609086:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204788.609096:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443107456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204788.609103:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204788.609106:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204788.609107:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204788.609109:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4060 00000100:00080000:0.0:1788204788.609111:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204788.609113:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204788.609116:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204788.609124:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443107584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204788.609125:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443107328:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204788.609140:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204788.609142:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204788.609144:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204788.609146:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4060 00000100:00080000:0.0:1788204788.609148:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204788.609151:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204788.609153:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204788.609160:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443107712/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204788.609208:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204788.609278:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204788/real 1788204788] req@ffff9023450ba300 x1875064443107328/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204788.609301:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443107328/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204788.609312:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204788.609316:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204788.609321:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443107328:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204788.609328:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443107456:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204788.609349:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204788.609362:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204788/real 1788204788] req@ffff9023450b8000 x1875064443107456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204788.609374:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443107456/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204788.609382:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204788.609385:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204788.609388:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443107456:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204788.609415:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443107584:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204788.609433:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204788.609443:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204788/real 1788204788] req@ffff9023450b9180 x1875064443107584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204788.609454:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443107584/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204788.609462:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204788.609465:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204788.609468:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443107584:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204788.609473:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443107712:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204788.609488:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204788.609497:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204788/real 1788204788] req@ffff9023450baa00 x1875064443107712/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204788.609507:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443107712/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204788.609514:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204788.609517:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204788.609520:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443107712:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204789.441091:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204790.465072:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204791.489059:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204792.513071:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204793.537223:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204793.729004:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204793.729013:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204793.729018:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204793.729029:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4065 00000100:00080000:3.0:1788204793.729040:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204793.729046:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204793.729088:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443107840/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204793.729111:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204793.729114:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204793.729116:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204793.729119:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4065 00000100:00080000:3.0:1788204793.729121:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204793.729124:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204793.729132:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204793.729141:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443107968/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204793.729147:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204793.729149:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204793.729150:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204793.729153:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4065 00000100:00080000:3.0:1788204793.729154:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204793.729157:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204793.729159:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204793.729166:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443107840:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204793.729167:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443108096/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204793.729192:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204793.729195:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204793.729196:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204793.729198:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4065 00000100:00080000:3.0:1788204793.729201:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204793.729203:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204793.729206:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204793.729215:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443108224/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204793.729255:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204793.729296:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204793/real 1788204793] req@ffff90235268aa00 x1875064443107840/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204793.729316:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443107840/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204793.729326:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204793.729330:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204793.729334:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443107840:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204793.729341:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443107968:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204793.729358:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204793.729367:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204793/real 1788204793] req@ffff902352689880 x1875064443107968/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204793.729379:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443107968/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204793.729386:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204793.729388:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204793.729391:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443107968:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204793.729415:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443108096:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204793.729429:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204793.729437:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204793/real 1788204793] req@ffff90235268bb80 x1875064443108096/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204793.729455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443108096/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204793.729462:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204793.729465:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204793.729468:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443108096:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204793.729472:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443108224:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204793.729484:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204793.729492:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204793/real 1788204793] req@ffff902352688700 x1875064443108224/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204793.729502:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443108224/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204793.729510:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204793.729512:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204793.729515:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443108224:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788204794.561056:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204795.585043:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204796.609054:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204797.633079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204798.657061:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204798.848999:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204798.849012:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204798.849017:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204798.849030:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4070 00000100:00080000:3.0:1788204798.849044:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204798.849053:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204798.849122:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443108352/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204798.849149:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204798.849152:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204798.849153:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204798.849157:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4070 00000100:00080000:3.0:1788204798.849160:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204798.849162:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204798.849171:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204798.849181:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443108480/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204798.849187:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204798.849190:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204798.849192:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204798.849194:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4070 00000100:00080000:3.0:1788204798.849196:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204798.849202:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204798.849204:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443108352:192.168.201.130@tcp:250:kworker.0 00000008:00100000:3.0:1788204798.849205:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204798.849215:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443108608/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204798.849221:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204798.849223:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204798.849224:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204798.849226:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4070 00000100:00080000:3.0:1788204798.849228:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204798.849230:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204798.849233:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204798.849239:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443108736/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204798.849283:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204798.849328:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204798/real 1788204798] req@ffff902352689180 x1875064443108352/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204798.849348:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443108352/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204798.849359:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204798.849363:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204798.849367:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443108352:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204798.849375:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443108480:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204798.849396:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204798.849409:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204798/real 1788204798] req@ffff902352689500 x1875064443108480/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204798.849421:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443108480/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204798.849429:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204798.849432:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204798.849436:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443108480:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204798.849460:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443108608:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204798.849476:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204798.849483:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204798/real 1788204798] req@ffff90235268a680 x1875064443108608/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204798.849490:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443108608/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204798.849500:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204798.849502:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204798.849504:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443108608:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204798.849507:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443108736:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204798.849520:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204798.849528:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204798/real 1788204798] req@ffff902352688e00 x1875064443108736/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204798.849539:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443108736/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204798.849547:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204798.849550:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204798.849553:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443108736:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204799.681125:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204800.705109:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204801.729052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204802.753069:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204803.777039:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204803.968961:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204803.968968:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204803.968971:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204803.968979:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4075 00000100:00080000:3.0:1788204803.968987:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204803.968993:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204803.969099:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443108864/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204803.969121:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204803.969124:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204803.969126:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204803.969130:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4075 00000100:00080000:3.0:1788204803.969134:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204803.969138:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204803.969146:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204803.969159:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443108992/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204803.969170:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204803.969173:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204803.969176:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204803.969181:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4075 00000100:00100000:2.0:1788204803.969182:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443108864:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204803.969184:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204803.969191:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204803.969195:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204803.969215:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443109120/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204803.969224:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204803.969228:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204803.969229:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788204803.969233:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204803.969236:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4075 00000100:00080000:3.0:1788204803.969241:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204803.969246:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204803.969251:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204803.969267:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204803/real 1788204803] req@ffff902352689c00 x1875064443108864/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204803.969270:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443109248/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204803.969293:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443108864/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204803.969301:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204803.969305:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204803.969308:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443108864:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204803.969330:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443108992:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204803.969344:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204803.969351:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204803/real 1788204803] req@ffff902352688000 x1875064443108992/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204803.969359:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443108992/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204803.969364:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204803.969366:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204803.969368:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443108992:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204803.969371:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443109120:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204803.969381:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204803.969387:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204803/real 1788204803] req@ffff90235268b800 x1875064443109120/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204803.969393:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443109120/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204803.969398:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204803.969399:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204803.969401:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443109120:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204803.969404:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443109248:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204803.969412:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204803.969418:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204803/real 1788204803] req@ffff90235268a300 x1875064443109248/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204803.969428:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443109248/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204803.969442:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204803.969445:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204803.969448:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443109248:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788204804.801094:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204805.825076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204806.849090:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204807.873048:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204808.897085:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204809.089018:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204809.089029:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204809.089037:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204809.089048:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4080 00000100:00080000:3.0:1788204809.089059:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204809.089066:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204809.089110:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064443109376/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204809.089147:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204809.089152:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204809.089153:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204809.089158:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4080 00000100:00080000:3.0:1788204809.089162:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204809.089166:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204809.089178:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204809.089195:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443109376:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204809.089195:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443109504/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204809.089208:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204809.089210:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204809.089212:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204809.089215:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4080 00000100:00080000:3.0:1788204809.089218:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204809.089222:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204809.089226:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204809.089239:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443109632/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204809.089258:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204809.089262:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204809.089264:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788204809.089268:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204809.089272:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4080 00000100:00080000:3.0:1788204809.089279:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204809.089285:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204809.089290:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204809.089303:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443109760/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204809.089312:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204809/real 1788204809] req@ffff90235268b100 x1875064443109376/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204809.089328:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064443109376/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204809.089337:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204809.089340:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204809.089343:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443109376:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204809.089364:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443109504:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204809.089379:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204809.089389:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204809/real 1788204809] req@ffff90235268a300 x1875064443109504/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204809.089397:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443109504/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204809.089403:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204809.089405:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204809.089407:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443109504:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204809.089412:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443109632:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204809.089426:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204809.089435:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204809/real 1788204809] req@ffff90235268b800 x1875064443109632/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204809.089444:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443109632/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204809.089450:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204809.089454:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204809.089456:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443109632:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204809.089460:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443109760:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204809.089474:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204809.089484:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204809/real 1788204809] req@ffff902352688000 x1875064443109760/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204809.089497:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443109760/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204809.089505:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204809.089509:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204809.089513:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443109760:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204809.921086:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204810.945104:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204811.969034:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204812.993046:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204814.016992:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204814.208958:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204814.208965:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204814.208968:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204814.208979:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4085 00000100:00080000:3.0:1788204814.208993:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204814.208998:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204814.209028:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443109888/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204814.209048:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204814.209051:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204814.209053:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204814.209057:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4085 00000100:00080000:3.0:1788204814.209060:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204814.209063:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204814.209072:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204814.209083:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443110016/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204814.209091:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443109888:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204814.209092:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204814.209095:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204814.209097:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204814.209100:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4085 00000100:00080000:3.0:1788204814.209103:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204814.209106:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204814.209110:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204814.209118:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443110144/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204814.209125:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204814.209127:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204814.209129:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204814.209131:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4085 00000100:00080000:3.0:1788204814.209133:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204814.209136:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204814.209139:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788204814.209142:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204814.209147:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443110272/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204814.209183:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204814/real 1788204814] req@ffff902352689c00 x1875064443109888/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204814.209200:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443109888/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204814.209207:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204814.209209:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204814.209212:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443109888:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204814.209229:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443110016:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204814.209245:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204814.209253:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204814/real 1788204814] req@ffff902352688e00 x1875064443110016/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204814.209259:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443110016/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204814.209264:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204814.209266:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204814.209268:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443110016:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204814.209271:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443110144:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204814.209282:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204814.209287:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204814/real 1788204814] req@ffff90235268a680 x1875064443110144/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204814.209293:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443110144/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204814.209297:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204814.209299:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204814.209301:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443110144:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204814.209304:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443110272:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204814.209312:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204814.209317:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204814/real 1788204814] req@ffff902352689500 x1875064443110272/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204814.209323:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443110272/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204814.209327:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204814.209328:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204814.209330:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443110272:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788204815.041019:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204816.065022:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204817.089075:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204818.113030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204819.137002:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204819.328971:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204819.328979:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204819.328982:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204819.328991:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4090 00000100:00080000:3.0:1788204819.328999:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204819.329004:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204819.329034:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443110400/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204819.329057:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204819.329059:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204819.329061:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204819.329064:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4090 00000100:00080000:3.0:1788204819.329068:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204819.329071:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204819.329079:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204819.329093:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443110528/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204819.329104:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204819.329107:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204819.329111:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788204819.329113:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443110400:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204819.329116:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4090 00000100:00080000:3.0:1788204819.329130:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204819.329133:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204819.329138:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204819.329148:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443110656/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204819.329155:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204819.329158:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204819.329161:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204819.329164:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4090 00000100:00080000:3.0:1788204819.329168:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788204819.329171:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204819.329174:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204819.329181:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204819.329193:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443110784/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204819.329210:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204819/real 1788204819] req@ffff902352689180 x1875064443110400/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204819.329240:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443110400/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204819.329251:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204819.329255:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204819.329259:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443110400:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204819.329279:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443110528:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204819.329300:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204819.329309:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204819/real 1788204819] req@ffff902352688700 x1875064443110528/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204819.329317:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443110528/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204819.329323:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204819.329325:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204819.329327:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443110528:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204819.329331:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443110656:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204819.329341:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204819.329347:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204819/real 1788204819] req@ffff90235268bb80 x1875064443110656/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204819.329354:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443110656/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204819.329359:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204819.329361:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204819.329363:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443110656:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204819.329366:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443110784:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204819.329375:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204819.329380:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204819/real 1788204819] req@ffff902352689880 x1875064443110784/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204819.329392:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443110784/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204819.329396:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204819.329398:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204819.329400:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443110784:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204820.161042:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204821.184999:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204822.209048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204823.233022:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204824.257018:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204824.448964:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204824.448979:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204824.448982:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204824.448990:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4095 00000100:00080000:3.0:1788204824.448998:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204824.449003:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204824.449031:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443110912/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204824.449053:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204824.449056:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204824.449057:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204824.449060:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4095 00000100:00080000:3.0:1788204824.449062:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204824.449066:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204824.449075:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204824.449088:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064443111040/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204824.449093:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443110912:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204824.449105:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204824.449108:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204824.449110:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204824.449114:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4095 00000100:00080000:3.0:1788204824.449117:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204824.449121:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204824.449127:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204824.449140:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443111168/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204824.449148:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204824.449160:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204824.449164:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204824.449173:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788204824.449180:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204824/real 1788204824] req@ffff90235268aa00 x1875064443110912/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204824.449181:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4095 00000100:00080000:3.0:1788204824.449187:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204824.449195:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204824.449202:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204824.449205:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443110912/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204824.449213:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788204824.449216:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443111296/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204824.449218:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204824.449224:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443110912:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204824.449246:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443111040:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204824.449260:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204824.449268:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204824/real 1788204824] req@ffff902352688a80 x1875064443111040/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204824.449275:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064443111040/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204824.449280:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204824.449281:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204824.449283:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443111040:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204824.449286:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443111168:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204824.449296:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204824.449304:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204824/real 1788204824] req@ffff90235268ad80 x1875064443111168/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204824.449314:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443111168/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204824.449328:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204824.449332:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204824.449335:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443111168:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204824.449340:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443111296:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204824.449358:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204824.449369:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204824/real 1788204824] req@ffff90235268b480 x1875064443111296/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204824.449380:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443111296/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204824.449386:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204824.449389:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204824.449392:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443111296:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788204825.281047:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204826.305019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204827.329019:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204828.353109:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204829.376992:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204829.569009:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204829.569022:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204829.569027:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204829.569037:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4101 00000100:00080000:3.0:1788204829.569049:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204829.569055:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204829.569096:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443111424/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204829.569123:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204829.569127:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204829.569129:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204829.569134:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4101 00000100:00080000:3.0:1788204829.569137:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204829.569140:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204829.569151:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204829.569167:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443111552/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204829.569176:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204829.569179:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204829.569180:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204829.569183:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4101 00000100:00080000:3.0:1788204829.569186:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788204829.569192:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443111424:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204829.569194:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204829.569203:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204829.569218:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443111680/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204829.569229:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204829.569232:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204829.569234:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204829.569238:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4101 00000100:00080000:3.0:1788204829.569254:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204829.569259:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204829.569264:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788204829.569266:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204829.569278:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064443111808/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204829.569318:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204829/real 1788204829] req@ffff902352688380 x1875064443111424/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204829.569344:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443111424/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204829.569356:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204829.569359:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204829.569362:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443111424:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204829.569368:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443111552:192.168.201.130@tcp:38:kworker.0 00000800:00000100:2.0:1788204829.569385:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204829.569394:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204829/real 1788204829] req@ffff90235268b480 x1875064443111552/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204829.569401:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443111552/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204829.569413:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204829.569415:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204829.569418:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443111552:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204829.569439:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443111680:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204829.569451:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204829.569458:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204829/real 1788204829] req@ffff90235268ad80 x1875064443111680/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204829.569465:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443111680/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204829.569470:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204829.569472:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204829.569473:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443111680:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204829.569477:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443111808:192.168.201.130@tcp:8:kworker.0 00000800:00000100:2.0:1788204829.569494:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204829.569500:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204829/real 1788204829] req@ffff902352688a80 x1875064443111808/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204829.569508:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064443111808/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204829.569515:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204829.569519:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204829.569523:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443111808:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788204830.401072:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204831.424983:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204832.449006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204833.473008:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204834.497019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204834.688976:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204834.688983:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204834.688986:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204834.688994:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4106 00000100:00080000:1.0:1788204834.689002:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204834.689007:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204834.689040:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064443111936/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204834.689065:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204834.689069:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204834.689070:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204834.689073:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4106 00000100:00080000:1.0:1788204834.689076:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204834.689079:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204834.689086:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204834.689097:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443112064/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204834.689104:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204834.689108:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204834.689109:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204834.689115:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4106 00000100:00100000:2.0:1788204834.689116:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443111936:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204834.689128:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204834.689137:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204834.689142:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204834.689155:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443112192/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204834.689164:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204834.689167:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204834.689168:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204834.689171:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4106 00000100:00080000:1.0:1788204834.689174:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204834.689177:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204834.689180:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204834.689188:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443112320/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204834.689220:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443112064:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788204834.689251:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443112192:192.168.201.130@tcp:8:kworker.0 00000100:00100000:2.0:1788204834.689274:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443112320:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788204835.521010:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204836.545639:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204837.569039:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788204837.761169:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:1.0:1788204837.761215:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788204837.761233:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204837.761282:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204837.761292:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204837.761302:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788204837.761388:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204834/real 1788204837] req@ffff90234519e300 x1875064443112064/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204837.761413:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443112064/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204837.761436:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204837.761441:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204837.761445:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443112064:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204837.761452:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204834/real 1788204837] req@ffff90234519d880 x1875064443112192/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204837.761465:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443112192/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204837.761472:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204837.761477:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204837.761480:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443112192:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204837.761487:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204834/real 1788204837] req@ffff90234519ed80 x1875064443112320/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204837.761499:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443112320/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204837.761506:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204837.761510:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204837.761513:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443112320:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204837.761519:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204834/real 1788204837] req@ffff90234519f800 x1875064443111936/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204837.761532:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064443111936/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204837.761552:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204837.761556:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204837.761559:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443111936:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788204838.593092:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204839.617200:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204839.809172:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204839.809186:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204839.809191:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204839.809203:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4111 00000100:00080000:3.0:1788204839.809217:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204839.809224:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204839.809267:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443112448/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204839.809300:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204839.809304:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204839.809306:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204839.809312:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4111 00000100:00080000:3.0:1788204839.809316:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204839.809320:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204839.809332:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204839.809349:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443112576/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204839.809357:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443112448:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204839.809363:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204839.809368:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204839.809370:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204839.809375:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4111 00000100:00080000:3.0:1788204839.809378:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204839.809395:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204839.809401:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204839.809414:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443112704/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204839.809423:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204839.809427:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204839.809429:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204839.809432:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4111 00000100:00080000:3.0:1788204839.809438:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204839.809442:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204839.809447:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204839.809459:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443112832/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204839.809473:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204839.809532:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204839/real 1788204839] req@ffff90235268aa00 x1875064443112448/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204839.809554:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443112448/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204839.809565:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204839.809569:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204839.809574:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443112448:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204839.809604:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443112576:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204839.809627:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204839.809647:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204839/real 1788204839] req@ffff902352689880 x1875064443112576/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204839.809661:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443112576/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204839.809669:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204839.809673:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204839.809675:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443112576:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204839.809682:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443112704:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204839.809707:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204839.809717:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204839/real 1788204839] req@ffff90235268bb80 x1875064443112704/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204839.809728:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443112704/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204839.809742:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204839.809745:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204839.809747:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443112704:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204839.809753:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443112832:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204839.809776:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204839.809787:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204839/real 1788204839] req@ffff902352688700 x1875064443112832/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204839.809797:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443112832/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204839.809805:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204839.809808:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204839.809811:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443112832:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204840.641105:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204841.665061:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204842.689023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204843.713081:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204844.737029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204844.928988:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204844.928998:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204844.929003:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204844.929012:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4116 00000100:00080000:1.0:1788204844.929022:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204844.929030:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204844.929064:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443112960/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204844.929095:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204844.929099:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204844.929101:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204844.929105:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4116 00000100:00080000:1.0:1788204844.929109:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204844.929113:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204844.929134:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204844.929151:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443113088/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204844.929155:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443112960:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204844.929168:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204844.929171:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204844.929174:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204844.929179:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4116 00000100:00080000:1.0:1788204844.929183:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204844.929187:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204844.929192:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204844.929204:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443113216/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204844.929211:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788204844.929220:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204844.929224:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204844.929226:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204844.929230:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4116 00000100:00080000:1.0:1788204844.929234:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204844.929239:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204844.929243:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204844.929254:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443113344/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204844.929288:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204844/real 1788204844] req@ffff90234519df80 x1875064443112960/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204844.929324:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443112960/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204844.929331:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204844.929334:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204844.929336:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443112960:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204844.929451:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443113088:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204844.929510:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204844.929524:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204844/real 1788204844] req@ffff90234519fb80 x1875064443113088/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204844.929540:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443113088/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204844.929548:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204844.929552:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204844.929555:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443113088:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204844.929562:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443113216:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204844.929585:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204844.929595:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204844/real 1788204844] req@ffff90234519c700 x1875064443113216/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204844.929607:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443113216/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204844.929614:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204844.929618:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204844.929620:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443113216:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204844.929626:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443113344:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204844.929645:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204844.929656:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204844/real 1788204844] req@ffff90234519c000 x1875064443113344/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204844.929666:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443113344/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204844.929674:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204844.929677:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204844.929680:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443113344:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788204845.761098:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204846.785055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204847.809079:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204848.833048:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204849.857073:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204850.048996:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204850.049003:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204850.049006:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204850.049014:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4121 00000100:00080000:2.0:1788204850.049023:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204850.049030:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204850.049065:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443113472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204850.049097:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204850.049101:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204850.049103:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204850.049108:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4121 00000100:00080000:2.0:1788204850.049111:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204850.049116:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204850.049124:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204850.049136:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443113600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204850.049143:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204850.049146:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204850.049147:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204850.049149:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4121 00000100:00080000:2.0:1788204850.049152:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204850.049158:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443113472:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204850.049159:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204850.049168:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204850.049182:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443113728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204850.049192:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204850.049196:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204850.049198:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204850.049214:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4121 00000100:00080000:2.0:1788204850.049225:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204850.049228:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204850.049232:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788204850.049235:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204850.049246:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443113856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204850.049270:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204850/real 1788204850] req@ffff90234598f480 x1875064443113472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204850.049288:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443113472/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204850.049295:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204850.049297:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204850.049300:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443113472:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204850.049305:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443113600:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204850.049324:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204850.049335:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204850/real 1788204850] req@ffff90234598ce00 x1875064443113600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204850.049347:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443113600/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204850.049354:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204850.049357:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204850.049359:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443113600:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204850.049378:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443113728:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204850.049392:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204850.049399:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204850/real 1788204850] req@ffff90234598f800 x1875064443113728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204850.049406:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443113728/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204850.049415:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204850.049417:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204850.049419:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443113728:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204850.049421:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443113856:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204850.049435:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204850.049444:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204850/real 1788204850] req@ffff90234598dc00 x1875064443113856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204850.049454:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443113856/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204850.049462:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204850.049465:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204850.049468:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443113856:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204850.881025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204851.905019:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204852.929012:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204853.953064:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204854.977082:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204855.169036:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204855.169048:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204855.169052:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204855.169062:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4126 00000100:00080000:0.0:1788204855.169073:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204855.169080:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204855.169139:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443113984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204855.169167:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204855.169170:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204855.169171:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204855.169174:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4126 00000100:00080000:0.0:1788204855.169177:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204855.169181:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204855.169190:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204855.169202:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443114112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204855.169208:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204855.169211:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204855.169212:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204855.169217:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4126 00000100:00080000:0.0:1788204855.169219:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204855.169221:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204855.169226:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204855.169226:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443113984:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204855.169245:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443114240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204855.169254:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204855.169258:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204855.169260:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204855.169264:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4126 00000100:00080000:0.0:1788204855.169268:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204855.169273:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204855.169277:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204855.169285:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443114368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204855.169303:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204855.169342:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204855/real 1788204855] req@ffff9023450bad80 x1875064443113984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204855.169355:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443113984/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204855.169363:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204855.169366:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204855.169369:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443113984:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204855.169392:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443114112:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204855.169409:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204855.169420:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204855/real 1788204855] req@ffff9023450b8700 x1875064443114112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204855.169433:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443114112/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204855.169450:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204855.169453:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204855.169455:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443114112:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204855.169460:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443114240:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204855.169475:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204855.169483:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204855/real 1788204855] req@ffff9023450b9f80 x1875064443114240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204855.169490:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443114240/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204855.169495:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204855.169497:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204855.169499:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443114240:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204855.169502:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443114368:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204855.169511:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204855.169516:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204855/real 1788204855] req@ffff9023450b9c00 x1875064443114368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204855.169523:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443114368/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204855.169527:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204855.169529:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204855.169537:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443114368:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204856.001038:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204857.025072:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204858.049012:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204859.073031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204860.097010:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204860.289009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204860.289019:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204860.289023:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204860.289043:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4131 00000100:00080000:2.0:1788204860.289061:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204860.289068:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204860.289105:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064443114496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204860.289132:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204860.289136:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204860.289137:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204860.289141:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4131 00000100:00080000:2.0:1788204860.289145:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204860.289150:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204860.289159:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204860.289172:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443114496:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204860.289173:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064443114624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204860.289187:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204860.289190:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204860.289192:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204860.289196:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4131 00000100:00080000:2.0:1788204860.289200:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204860.289204:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204860.289209:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204860.289220:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064443114752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204860.289225:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788204860.289235:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204860.289239:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204860.289249:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204860.289257:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4131 00000100:00100000:3.0:1788204860.289258:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204860/real 1788204860] req@ffff90234598ca80 x1875064443114496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204860.289264:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204860.289272:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204860.289276:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064443114496/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788204860.289277:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1788204860.289290:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204860.289293:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:2.0:1788204860.289297:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443114880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204860.289299:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443114496:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204860.289327:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443114624:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204860.289351:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204860.289363:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204860/real 1788204860] req@ffff90234598e680 x1875064443114624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204860.289374:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064443114624/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204860.289388:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204860.289390:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204860.289392:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443114624:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204860.289396:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443114752:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204860.289411:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204860.289418:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204860/real 1788204860] req@ffff90234598d880 x1875064443114752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204860.289424:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064443114752/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204860.289429:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204860.289431:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204860.289433:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443114752:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204860.289436:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443114880:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204860.289444:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204860.289449:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204860/real 1788204860] req@ffff90234598dc00 x1875064443114880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204860.289455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443114880/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204860.289459:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204860.289461:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204860.289463:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443114880:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204861.121062:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204862.145033:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204863.169033:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204864.193031:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204865.217070:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204865.409008:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204865.409018:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204865.409022:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204865.409031:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4136 00000100:00080000:2.0:1788204865.409049:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204865.409056:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204865.409090:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443115008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204865.409124:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204865.409128:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204865.409129:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204865.409133:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4136 00000100:00080000:2.0:1788204865.409137:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204865.409141:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204865.409151:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204865.409164:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443115136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204865.409170:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443115008:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204865.409180:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204865.409184:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204865.409185:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204865.409190:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4136 00000100:00080000:2.0:1788204865.409193:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204865.409196:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204865.409201:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204865.409213:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443115264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204865.409222:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204865.409225:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204865.409227:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788204865.409232:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788204865.409237:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4136 00000100:00080000:2.0:1788204865.409247:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204865.409251:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204865.409256:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204865.409268:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443115392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204865.409279:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204865/real 1788204865] req@ffff90234598f800 x1875064443115008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204865.409296:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443115008/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204865.409306:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204865.409310:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204865.409314:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443115008:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204865.409335:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443115136:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204865.409353:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204865.409362:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204865/real 1788204865] req@ffff90234598ce00 x1875064443115136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204865.409372:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443115136/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204865.409379:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204865.409382:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204865.409386:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443115136:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204865.409391:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443115264:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204865.409408:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204865.409415:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204865/real 1788204865] req@ffff90234598f480 x1875064443115264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204865.409422:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443115264/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204865.409427:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204865.409429:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204865.409431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443115264:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204865.409434:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443115392:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204865.409443:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204865.409448:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204865/real 1788204865] req@ffff90234598c700 x1875064443115392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204865.409454:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443115392/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204865.409458:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204865.409460:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204865.409461:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443115392:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204866.241023:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204867.265039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204868.289045:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204869.313023:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204870.336990:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204870.528992:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204870.529002:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204870.529006:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204870.529015:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4142 00000100:00080000:2.0:1788204870.529025:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204870.529031:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204870.529065:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443115520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204870.529085:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204870.529088:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204870.529089:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204870.529092:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4142 00000100:00080000:2.0:1788204870.529095:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204870.529097:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204870.529104:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204870.529112:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443115648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204870.529120:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204870.529122:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204870.529123:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788204870.529127:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443115520:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204870.529128:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4142 00000100:00080000:2.0:1788204870.529132:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204870.529135:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204870.529139:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204870.529147:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443115776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204870.529152:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204870.529154:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204870.529155:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204870.529157:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4142 00000100:00080000:2.0:1788204870.529160:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204870.529162:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204870.529165:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204870.529171:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064443115904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204870.529185:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204870.529216:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204870/real 1788204870] req@ffff90234598c380 x1875064443115520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204870.529227:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443115520/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204870.529249:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204870.529253:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204870.529258:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443115520:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204870.529279:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443115648:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204870.529299:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204870.529310:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204870/real 1788204870] req@ffff90234598d180 x1875064443115648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204870.529320:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443115648/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204870.529327:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204870.529330:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204870.529333:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443115648:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204870.529338:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443115776:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204870.529352:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204870.529360:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204870/real 1788204870] req@ffff90234598e300 x1875064443115776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204870.529370:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443115776/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204870.529377:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204870.529379:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204870.529382:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443115776:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204870.529387:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443115904:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204870.529400:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204870.529408:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204870/real 1788204870] req@ffff90234598f100 x1875064443115904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204870.529417:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064443115904/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204870.529424:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204870.529427:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204870.529429:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443115904:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204871.361034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204872.384988:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204873.409071:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204874.432989:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204875.457039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204875.648959:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204875.648968:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204875.648972:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204875.648980:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4147 00000100:00080000:0.0:1788204875.648989:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204875.648994:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204875.649023:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443116032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204875.649043:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204875.649045:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204875.649047:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204875.649049:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4147 00000100:00080000:0.0:1788204875.649051:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204875.649053:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204875.649060:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204875.649073:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443116160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204875.649081:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204875.649084:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204875.649085:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204875.649087:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4147 00000100:00080000:0.0:1788204875.649090:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204875.649092:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204875.649095:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204875.649101:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443116288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204875.649108:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443116032:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204875.649110:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204875.649115:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204875.649118:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204875.649121:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4147 00000100:00080000:0.0:1788204875.649122:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204875.649124:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204875.649132:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204875.649139:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443116416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204875.649173:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204875.649208:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204875/real 1788204875] req@ffff9023450b8380 x1875064443116032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204875.649226:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443116032/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204875.649237:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204875.649241:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204875.649245:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443116032:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204875.649252:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443116160:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204875.649273:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204875.649285:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204875/real 1788204875] req@ffff9023450b8a80 x1875064443116160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204875.649297:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443116160/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204875.649304:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204875.649307:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204875.649311:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443116160:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204875.649332:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443116288:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204875.649350:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204875.649366:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204875/real 1788204875] req@ffff9023450bb800 x1875064443116288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204875.649379:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443116288/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204875.649395:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204875.649399:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204875.649402:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443116288:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204875.649408:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443116416:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204875.649425:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204875.649435:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204875/real 1788204875] req@ffff9023450b9500 x1875064443116416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204875.649445:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443116416/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204875.649458:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204875.649461:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204875.649464:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443116416:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204876.481023:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204877.505004:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204878.529034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204879.552992:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204880.577013:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204880.768970:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204880.768978:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204880.768980:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204880.768989:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4152 00000100:00080000:0.0:1788204880.768999:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204880.769005:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204880.769038:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443116544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204880.769061:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204880.769064:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204880.769070:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204880.769073:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4152 00000100:00080000:0.0:1788204880.769077:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204880.769080:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204880.769088:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204880.769097:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064443116672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204880.769102:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443116544:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788204880.769106:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204880.769109:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204880.769111:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204880.769114:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4152 00000100:00080000:0.0:1788204880.769115:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204880.769117:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204880.769123:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204880.769132:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064443116800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204880.769138:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204880.769141:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204880.769142:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204880.769143:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4152 00000100:00080000:0.0:1788204880.769145:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204880.769147:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204880.769150:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204880.769156:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064443116928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204880.769164:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204880.769200:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204880/real 1788204880] req@ffff9023450bb480 x1875064443116544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204880.769217:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443116544/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204880.769227:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204880.769230:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204880.769235:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443116544:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204880.769269:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443116672:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204880.769295:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204880.769311:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204880/real 1788204880] req@ffff9023450bbb80 x1875064443116672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204880.769323:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064443116672/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204880.769332:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204880.769336:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204880.769342:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443116672:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204880.769349:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443116800:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204880.769373:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204880.769385:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204880/real 1788204880] req@ffff9023450b8e00 x1875064443116800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204880.769393:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064443116800/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204880.769400:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204880.769403:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204880.769469:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443116800:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204880.769479:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443116928:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204880.769502:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204880.769511:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204880/real 1788204880] req@ffff9023450ba680 x1875064443116928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204880.769518:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064443116928/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204880.769523:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204880.769525:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204880.769527:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443116928:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204881.601031:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204882.625022:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204883.649022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204884.673019:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204885.696997:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204885.888995:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204885.889007:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204885.889011:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204885.889031:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4157 00000100:00080000:3.0:1788204885.889042:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204885.889049:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204885.889084:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443117056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204885.889112:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204885.889116:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204885.889118:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204885.889123:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4157 00000100:00080000:3.0:1788204885.889127:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204885.889132:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204885.889140:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443117056:192.168.201.130@tcp:250:kworker.0 00000008:00100000:3.0:1788204885.889142:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204885.889160:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443117184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204885.889172:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204885.889175:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204885.889177:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204885.889180:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4157 00000100:00080000:3.0:1788204885.889187:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204885.889191:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204885.889197:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204885.889198:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204885.889212:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443117312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204885.889228:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204885.889231:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204885.889233:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204885.889238:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4157 00000100:00100000:0.0:1788204885.889239:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204885/real 1788204885] req@ffff902352689180 x1875064443117056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204885.889244:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204885.889252:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204885.889258:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204885.889260:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443117056/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204885.889269:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788204885.889272:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443117440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204885.889274:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204885.889281:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443117056:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204885.889305:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443117184:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204885.889325:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204885.889335:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204885/real 1788204885] req@ffff902352689500 x1875064443117184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204885.889347:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443117184/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204885.889355:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204885.889357:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204885.889360:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443117184:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204885.889365:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443117312:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204885.889382:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204885.889392:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204885/real 1788204885] req@ffff90235268a680 x1875064443117312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204885.889400:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443117312/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204885.889416:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204885.889418:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204885.889423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443117312:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204885.889426:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443117440:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204885.889439:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204885.889445:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204885/real 1788204885] req@ffff902352688e00 x1875064443117440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204885.889452:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443117440/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204885.889456:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204885.889458:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204885.889460:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443117440:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788204886.721041:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204887.745012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204888.769021:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204889.792988:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204890.817024:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204891.008962:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204891.008970:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204891.008973:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204891.008981:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4162 00000100:00080000:0.0:1788204891.008989:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204891.008994:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204891.009032:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064443117568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204891.009052:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204891.009055:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204891.009056:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204891.009058:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4162 00000100:00080000:0.0:1788204891.009061:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204891.009064:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204891.009073:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204891.009084:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443117568:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204891.009088:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064443117696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204891.009100:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204891.009104:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204891.009105:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204891.009109:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4162 00000100:00080000:0.0:1788204891.009113:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204891.009117:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204891.009121:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204891.009131:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064443117824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204891.009135:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788204891.009143:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204891.009146:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204891.009148:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204891.009151:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4162 00000100:00080000:0.0:1788204891.009154:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204891.009156:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204891.009160:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204891.009168:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443117952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204891.009171:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204891/real 1788204891] req@ffff9023450ba680 x1875064443117568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204891.009189:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064443117568/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204891.009196:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204891.009199:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204891.009201:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443117568:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204891.009217:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443117696:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204891.009230:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204891.009239:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204891/real 1788204891] req@ffff9023450b8e00 x1875064443117696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204891.009251:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064443117696/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204891.009259:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204891.009262:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204891.009265:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443117696:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204891.009271:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443117824:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204891.009289:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204891.009296:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204891/real 1788204891] req@ffff9023450bbb80 x1875064443117824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204891.009303:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064443117824/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204891.009308:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204891.009310:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204891.009312:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443117824:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204891.009315:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443117952:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204891.009325:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204891.009330:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204891/real 1788204891] req@ffff9023450bb480 x1875064443117952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204891.009336:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443117952/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204891.009344:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204891.009353:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204891.009358:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443117952:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204891.841012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204892.864990:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204893.888989:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204894.913025:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204895.937064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204896.129030:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204896.129041:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204896.129045:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204896.129055:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4167 00000100:00080000:0.0:1788204896.129067:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204896.129075:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204896.129117:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443118080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204896.129145:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204896.129148:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204896.129149:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204896.129152:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4167 00000100:00080000:0.0:1788204896.129154:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204896.129157:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204896.129166:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204896.129177:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443118208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204896.129185:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204896.129188:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204896.129189:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204896.129192:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4167 00000100:00080000:0.0:1788204896.129195:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204896.129198:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204896.129202:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204896.129209:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443118336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204896.129215:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204896.129218:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204896.129219:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204896.129222:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4167 00000100:00080000:0.0:1788204896.129224:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204896.129228:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204896.129232:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204896.129237:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443118080:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204896.129241:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443118464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204896.129326:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204896.129381:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204896/real 1788204896] req@ffff9023450b9500 x1875064443118080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204896.129400:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443118080/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204896.129423:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204896.129427:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204896.129431:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443118080:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204896.129439:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443118208:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204896.129463:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204896.129476:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204896/real 1788204896] req@ffff9023450bb800 x1875064443118208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204896.129489:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443118208/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204896.129497:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204896.129500:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204896.129503:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443118208:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204896.129510:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443118336:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204896.129530:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204896.129540:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204896/real 1788204896] req@ffff9023450b8a80 x1875064443118336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204896.129552:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443118336/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204896.129560:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204896.129563:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204896.129566:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443118336:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204896.129591:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443118464:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204896.129609:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204896.129618:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204896/real 1788204896] req@ffff9023450b8380 x1875064443118464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204896.129629:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443118464/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204896.129637:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204896.129640:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204896.129643:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443118464:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204896.961076:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204897.985020:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204899.009024:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204900.033030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204901.057039:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204901.248984:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204901.248993:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204901.248997:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204901.249005:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4172 00000100:00080000:0.0:1788204901.249013:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204901.249019:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204901.249051:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443118592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204901.249074:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204901.249079:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204901.249080:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204901.249083:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4172 00000100:00080000:0.0:1788204901.249085:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204901.249087:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204901.249094:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204901.249103:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443118720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204901.249111:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204901.249113:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204901.249114:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204901.249117:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4172 00000100:00080000:0.0:1788204901.249119:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204901.249121:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204901.249125:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204901.249132:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443118592:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204901.249134:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443118848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204901.249148:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204901.249152:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204901.249154:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204901.249159:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4172 00000100:00080000:0.0:1788204901.249163:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204901.249166:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788204901.249172:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204901.249186:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443118976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204901.249245:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443118720:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204901.249293:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443118848:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204901.249342:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443118976:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204902.081040:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204903.105141:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204904.129019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788204904.321149:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:1.0:1788204904.321190:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788204904.321207:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204904.321253:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204904.321275:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788204904.321285:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788204904.321377:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204901/real 1788204904] req@ffff9023450b8700 x1875064443118848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204904.321405:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443118848/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204904.321420:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204904.321427:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204904.321432:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443118848:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204904.321441:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204901/real 1788204904] req@ffff9023450bad80 x1875064443118976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204904.321454:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443118976/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204904.321461:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204904.321465:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204904.321468:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443118976:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204904.321475:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204901/real 1788204904] req@ffff9023450b9c00 x1875064443118592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204904.321489:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443118592/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204904.321505:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204904.321509:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204904.321513:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443118592:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204904.321520:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204901/real 1788204904] req@ffff9023450b9f80 x1875064443118720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204904.321534:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443118720/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204904.321542:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204904.321546:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204904.321549:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443118720:192.168.201.130@tcp:38:kworker.0 00010000:00010000:1.0:1788204905.153091:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204906.177046:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204906.369047:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204906.369065:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204906.369069:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204906.369080:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4177 00000100:00080000:1.0:1788204906.369092:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204906.369099:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204906.369136:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443119104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204906.369164:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204906.369168:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204906.369170:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204906.369174:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4177 00000100:00080000:1.0:1788204906.369183:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204906.369188:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204906.369198:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204906.369216:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064443119232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204906.369216:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443119104:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204906.369247:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204906.369251:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204906.369253:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204906.369257:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4177 00000100:00080000:1.0:1788204906.369262:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204906.369266:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204906.369272:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204906.369286:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443119360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204906.369297:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788204906.369303:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788204906.369304:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204906.369309:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204906.369315:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4177 00000100:00080000:1.0:1788204906.369319:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204906.369325:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204906.369330:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204906.369344:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443119488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204906.369351:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204906/real 1788204906] req@ffff90234519ea00 x1875064443119104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204906.369378:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443119104/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204906.369391:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204906.369396:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204906.369400:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443119104:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204906.369425:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443119232:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788204906.369452:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204906.369466:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204906/real 1788204906] req@ffff90234519dc00 x1875064443119232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204906.369480:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064443119232/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204906.369490:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204906.369493:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204906.369497:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443119232:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788204906.369504:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443119360:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204906.369527:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204906.369540:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204906/real 1788204906] req@ffff90234519f100 x1875064443119360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204906.369553:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443119360/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204906.369563:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204906.369566:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204906.369570:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443119360:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788204906.369576:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443119488:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788204906.369597:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204906.369609:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204906/real 1788204906] req@ffff90234519c380 x1875064443119488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204906.369627:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443119488/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204906.369635:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204906.369638:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788204906.369641:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443119488:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204907.201063:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204908.225046:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204909.249004:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204910.273102:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204911.297032:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204911.489006:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204911.489014:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204911.489017:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204911.489026:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4183 00000100:00080000:3.0:1788204911.489036:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204911.489041:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204911.489079:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443119616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204911.489107:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204911.489110:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204911.489111:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204911.489114:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4183 00000100:00080000:3.0:1788204911.489117:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204911.489119:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204911.489127:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204911.489138:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443119744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204911.489162:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443119616:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204911.489178:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204911.489181:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204911.489182:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204911.489186:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4183 00000100:00080000:3.0:1788204911.489190:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204911.489193:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204911.489197:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204911.489208:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443119872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204911.489215:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204911.489218:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204911.489219:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788204911.489224:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204911.489227:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4183 00000100:00080000:3.0:1788204911.489234:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204911.489239:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204911.489245:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204911.489262:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204911/real 1788204911] req@ffff902352689c00 x1875064443119616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204911.489262:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443120000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204911.489299:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443119616/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204911.489308:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204911.489311:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204911.489314:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443119616:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204911.489334:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443119744:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204911.489349:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204911.489357:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204911/real 1788204911] req@ffff902352688000 x1875064443119744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204911.489366:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443119744/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204911.489372:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204911.489373:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204911.489376:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443119744:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204911.489381:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443119872:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204911.489392:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204911.489398:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204911/real 1788204911] req@ffff90235268b800 x1875064443119872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204911.489406:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443119872/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204911.489411:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204911.489413:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204911.489415:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443119872:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204911.489418:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443120000:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204911.489428:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204911.489434:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204911/real 1788204911] req@ffff90235268a300 x1875064443120000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204911.489441:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443120000/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204911.489446:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204911.489448:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204911.489450:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443120000:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204912.321046:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204913.345033:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204914.369032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204915.393000:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204916.417072:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204916.609014:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204916.609035:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204916.609040:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204916.609068:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4188 00000100:00080000:3.0:1788204916.609080:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204916.609090:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204916.609133:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064443120128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204916.609166:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204916.609170:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204916.609173:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204916.609178:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4188 00000100:00080000:3.0:1788204916.609182:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204916.609188:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204916.609209:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204916.609218:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443120128:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204916.609231:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443120256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204916.609246:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204916.609250:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204916.609253:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204916.609259:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4188 00000100:00080000:3.0:1788204916.609265:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204916.609272:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204916.609280:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204916.609294:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204916.609309:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443120384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204916.609323:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204916.609328:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204916.609331:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204916.609337:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4188 00000100:00080000:3.0:1788204916.609343:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788204916.609345:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204916/real 1788204916] req@ffff90235268b100 x1875064443120128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204916.609351:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204916.609387:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064443120128/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204916.609399:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204916.609404:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204916.609409:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443120128:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204916.609436:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443120256:192.168.201.130@tcp:38:kworker.0 00000008:00100000:3.0:1788204916.609440:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204916.609461:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443120512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204916.609470:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204916.609490:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204916/real 1788204916] req@ffff90235268a300 x1875064443120256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204916.609510:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443120256/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204916.609519:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204916.609523:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204916.609527:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443120256:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204916.609534:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443120384:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204916.609553:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204916.609563:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204916/real 1788204916] req@ffff90235268b800 x1875064443120384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204916.609576:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443120384/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204916.609592:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204916.609596:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204916.609599:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443120384:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204916.609612:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443120512:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204916.609640:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204916.609652:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204916/real 1788204916] req@ffff902352688000 x1875064443120512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204916.609664:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443120512/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204916.609673:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204916.609676:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204916.609690:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443120512:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204917.441036:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204918.465000:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204919.489031:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204920.513023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204921.537074:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204921.729003:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204921.729012:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204921.729016:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204921.729025:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4193 00000100:00080000:1.0:1788204921.729034:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204921.729040:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204921.729073:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443120640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204921.729102:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204921.729106:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204921.729107:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204921.729111:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4193 00000100:00080000:1.0:1788204921.729114:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204921.729118:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204921.729125:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204921.729139:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443120640:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204921.729139:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443120768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204921.729159:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204921.729161:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204921.729163:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204921.729167:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4193 00000100:00080000:1.0:1788204921.729171:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204921.729175:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204921.729179:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204921.729191:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443120896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204921.729201:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788204921.729202:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204921.729207:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204921.729209:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204921.729214:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4193 00000100:00080000:1.0:1788204921.729218:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204921.729222:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204921.729228:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204921.729236:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204921/real 1788204921] req@ffff90234519f480 x1875064443120640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204921.729241:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064443121024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204921.729276:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443120640/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204921.729287:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204921.729290:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204921.729292:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443120640:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204921.729311:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443120768:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204921.729327:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204921.729336:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204921/real 1788204921] req@ffff90234519c380 x1875064443120768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204921.729343:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443120768/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204921.729348:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204921.729350:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204921.729352:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443120768:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204921.729355:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443120896:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204921.729365:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204921.729370:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204921/real 1788204921] req@ffff90234519f100 x1875064443120896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204921.729377:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443120896/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204921.729382:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204921.729383:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204921.729385:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443120896:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204921.729388:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443121024:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204921.729397:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204921.729402:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204921/real 1788204921] req@ffff90234519dc00 x1875064443121024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204921.729409:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064443121024/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204921.729413:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204921.729415:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204921.729416:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443121024:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204922.561136:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204923.585085:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204924.609062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204925.633036:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204926.657013:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204926.848947:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204926.848954:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204926.848957:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204926.848964:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4198 00000100:00080000:1.0:1788204926.848973:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204926.848978:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204926.849007:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443121152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204926.849028:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204926.849031:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204926.849032:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204926.849035:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4198 00000100:00080000:1.0:1788204926.849037:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204926.849040:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204926.849048:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204926.849062:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443121280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204926.849072:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204926.849075:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204926.849077:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204926.849082:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443121152:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204926.849084:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4198 00000100:00080000:1.0:1788204926.849091:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204926.849095:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204926.849099:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204926.849108:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443121408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204926.849115:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204926.849118:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204926.849119:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204926.849121:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4198 00000100:00080000:1.0:1788204926.849131:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204926.849133:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204926.849135:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204926.849143:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204926.849144:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443121536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204926.849182:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204926/real 1788204926] req@ffff90234519ea00 x1875064443121152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204926.849199:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443121152/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204926.849208:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204926.849212:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204926.849216:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443121152:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204926.849222:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443121280:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204926.849241:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204926.849253:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204926/real 1788204926] req@ffff90234519c000 x1875064443121280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204926.849264:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443121280/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204926.849278:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204926.849281:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204926.849284:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443121280:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204926.849303:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443121408:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204926.849319:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204926.849327:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204926/real 1788204926] req@ffff90234519c700 x1875064443121408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204926.849338:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443121408/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204926.849345:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204926.849347:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204926.849350:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443121408:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204926.849356:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443121536:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204926.849369:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204926.849377:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204926/real 1788204926] req@ffff90234519fb80 x1875064443121536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204926.849388:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443121536/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204926.849395:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204926.849398:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204926.849401:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443121536:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204927.680982:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204928.705025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204929.729020:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204930.753017:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204931.777035:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204931.968993:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204931.969003:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204931.969015:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204931.969025:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4203 00000100:00080000:3.0:1788204931.969035:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204931.969041:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204931.969075:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443121664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204931.969102:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204931.969106:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204931.969108:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204931.969111:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4203 00000100:00080000:3.0:1788204931.969115:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204931.969119:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204931.969128:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204931.969141:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443121792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204931.969150:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443121664:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204931.969155:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204931.969159:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204931.969161:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204931.969165:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4203 00000100:00080000:3.0:1788204931.969169:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204931.969172:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204931.969176:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204931.969187:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443121920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204931.969195:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204931.969200:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1788204931.969201:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788204931.969204:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204931.969210:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4203 00000100:00080000:3.0:1788204931.969215:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204931.969219:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204931.969231:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204931.969234:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204931/real 1788204931] req@ffff902352689c00 x1875064443121664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204931.969245:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443122048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204931.969249:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443121664/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204931.969270:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204931.969273:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204931.969277:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443121664:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204931.969294:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443121792:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204931.969307:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204931.969314:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204931/real 1788204931] req@ffff902352688e00 x1875064443121792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204931.969321:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443121792/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204931.969327:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204931.969328:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204931.969331:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443121792:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204931.969334:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443121920:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204931.969352:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204931.969359:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204931/real 1788204931] req@ffff90235268a680 x1875064443121920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204931.969366:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443121920/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204931.969371:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204931.969372:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204931.969374:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443121920:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204931.969379:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443122048:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204931.969394:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204931.969404:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204931/real 1788204931] req@ffff902352689500 x1875064443122048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204931.969415:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443122048/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204931.969423:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204931.969426:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204931.969429:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443122048:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204932.801009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204933.825025:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204934.848998:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204935.873008:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204936.897020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204937.088966:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204937.088974:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204937.088977:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204937.088985:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4208 00000100:00080000:3.0:1788204937.088993:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204937.088998:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204937.089028:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443122176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204937.089049:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204937.089051:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204937.089052:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204937.089055:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4208 00000100:00080000:3.0:1788204937.089057:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204937.089059:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204937.089065:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204937.089078:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443122304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204937.089085:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204937.089087:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204937.089088:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204937.089091:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4208 00000100:00080000:3.0:1788204937.089093:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204937.089095:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204937.089098:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204937.089101:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443122176:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788204937.089106:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443122432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204937.089116:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204937.089118:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204937.089119:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204937.089121:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4208 00000100:00080000:3.0:1788204937.089123:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204937.089124:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204937.089127:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204937.089134:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443122560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204937.089173:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204937.089208:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204937/real 1788204937] req@ffff902352689180 x1875064443122176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204937.089237:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443122176/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204937.089247:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204937.089250:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204937.089254:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443122176:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204937.089262:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443122304:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204937.089284:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204937.089296:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204937/real 1788204937] req@ffff902352688700 x1875064443122304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204937.089307:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443122304/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204937.089313:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204937.089316:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204937.089318:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443122304:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204937.089340:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443122432:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204937.089358:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204937.089368:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204937/real 1788204937] req@ffff90235268bb80 x1875064443122432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204937.089385:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443122432/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204937.089392:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204937.089394:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204937.089397:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443122432:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204937.089403:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443122560:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204937.089420:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204937.089431:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204937/real 1788204937] req@ffff902352689880 x1875064443122560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204937.089441:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443122560/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204937.089448:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204937.089451:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204937.089454:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443122560:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204937.921007:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204938.945010:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204939.969015:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204940.993058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204942.016991:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204942.208984:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204942.208994:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204942.208998:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204942.209008:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4213 00000100:00080000:1.0:1788204942.209018:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204942.209023:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204942.209053:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443122688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204942.209078:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204942.209081:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204942.209084:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204942.209087:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4213 00000100:00080000:1.0:1788204942.209090:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204942.209092:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204942.209098:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204942.209107:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064443122816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204942.209115:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788204942.209117:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443122688:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204942.209118:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204942.209121:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204942.209126:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4213 00000100:00080000:1.0:1788204942.209128:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204942.209130:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204942.209133:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204942.209141:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443122944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204942.209147:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204942.209149:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204942.209150:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204942.209153:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4213 00000100:00080000:1.0:1788204942.209161:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204942.209163:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204942.209166:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204942.209174:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204942.209178:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443123072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204942.209215:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204942/real 1788204942] req@ffff90234519df80 x1875064443122688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204942.209228:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443122688/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204942.209235:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204942.209239:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204942.209253:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443122688:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204942.209274:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443122816:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204942.209293:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204942.209302:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204942/real 1788204942] req@ffff90234519f800 x1875064443122816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204942.209310:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064443122816/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204942.209315:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204942.209317:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204942.209319:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443122816:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204942.209322:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443122944:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204942.209334:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204942.209343:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204942/real 1788204942] req@ffff90234519ed80 x1875064443122944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204942.209359:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443122944/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204942.209367:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204942.209370:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204942.209373:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443122944:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204942.209379:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443123072:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204942.209398:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204942.209409:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204942/real 1788204942] req@ffff90234519d880 x1875064443123072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204942.209420:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443123072/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204942.209429:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204942.209431:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204942.209433:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443123072:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204943.041017:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204944.065027:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204945.089034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204946.113022:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204947.137042:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204947.328991:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204947.329002:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204947.329006:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204947.329016:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4218 00000100:00080000:1.0:1788204947.329026:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204947.329033:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204947.329082:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443123200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204947.329111:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204947.329115:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204947.329117:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204947.329120:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4218 00000100:00080000:1.0:1788204947.329123:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204947.329126:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204947.329133:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204947.329145:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443123328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204947.329152:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204947.329154:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204947.329156:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204947.329160:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4218 00000100:00100000:0.0:1788204947.329161:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443123200:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204947.329162:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204947.329168:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204947.329173:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204947.329180:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443123456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204947.329186:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204947.329188:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204947.329189:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204947.329191:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4218 00000100:00080000:1.0:1788204947.329193:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204947.329195:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204947.329197:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204947.329203:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064443123584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204947.329230:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204947.329263:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204947/real 1788204947] req@ffff90234519e300 x1875064443123200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204947.329275:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443123200/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204947.329282:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204947.329284:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204947.329287:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443123200:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204947.329303:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443123328:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204947.329318:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204947.329326:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204947/real 1788204947] req@ffff90234519d880 x1875064443123328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204947.329332:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443123328/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204947.329337:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204947.329339:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204947.329341:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443123328:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204947.329344:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443123456:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204947.329354:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204947.329359:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204947/real 1788204947] req@ffff90234519ed80 x1875064443123456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204947.329365:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443123456/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204947.329370:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204947.329372:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204947.329373:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443123456:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204947.329376:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443123584:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204947.329384:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204947.329389:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204947/real 1788204947] req@ffff90234519f800 x1875064443123584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204947.329396:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064443123584/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204947.329400:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204947.329401:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204947.329403:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443123584:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204948.161029:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204949.185032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204950.209068:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204951.233040:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204952.257026:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204952.449023:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204952.449033:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204952.449038:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204952.449048:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4223 00000100:00080000:2.0:1788204952.449060:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204952.449066:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204952.449111:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443123712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204952.449146:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204952.449150:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204952.449152:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204952.449156:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4223 00000100:00080000:2.0:1788204952.449161:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204952.449165:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204952.449177:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204952.449188:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443123712:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204952.449192:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064443123840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204952.449204:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204952.449207:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204952.449210:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204952.449214:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4223 00000100:00080000:2.0:1788204952.449218:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204952.449223:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204952.449228:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204952.449241:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064443123968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204952.449249:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204952.449252:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204952.449254:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204952.449259:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4223 00000100:00080000:2.0:1788204952.449263:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204952.449267:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204952.449272:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788204952.449274:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788204952.449285:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064443124096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204952.449325:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204952/real 1788204952] req@ffff90234598df80 x1875064443123712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204952.449346:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443123712/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204952.449356:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204952.449359:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204952.449362:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443123712:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204952.449385:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443123840:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788204952.449404:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204952.449413:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204952/real 1788204952] req@ffff90234598fb80 x1875064443123840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204952.449420:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064443123840/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204952.449426:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204952.449428:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204952.449430:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443123840:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788204952.449457:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443123968:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204952.449472:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204952.449482:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204952/real 1788204952] req@ffff90234598ed80 x1875064443123968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204952.449493:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064443123968/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204952.449501:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204952.449504:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204952.449507:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443123968:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788204952.449512:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443124096:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204952.449531:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204952.449544:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204952/real 1788204952] req@ffff90234598f100 x1875064443124096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204952.449556:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064443124096/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204952.449565:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204952.449568:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204952.449571:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443124096:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204953.281085:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204954.305080:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204955.329064:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204956.353048:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204957.377049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204957.569050:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204957.569064:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204957.569070:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204957.569091:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4229 00000100:00080000:1.0:1788204957.569104:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204957.569112:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204957.569160:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443124224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204957.569196:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204957.569200:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204957.569202:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204957.569206:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4229 00000100:00080000:1.0:1788204957.569210:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204957.569215:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204957.569225:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204957.569240:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443124352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204957.569244:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443124224:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204957.569258:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204957.569262:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204957.569264:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204957.569270:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4229 00000100:00080000:1.0:1788204957.569273:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204957.569277:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204957.569282:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204957.569294:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443124480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204957.569304:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204957.569308:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204957.569310:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204957.569314:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4229 00000100:00080000:1.0:1788204957.569318:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788204957.569320:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788204957.569324:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204957.569330:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204957.569342:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443124608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204957.569381:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204957/real 1788204957] req@ffff90234519df80 x1875064443124224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204957.569403:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443124224/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204957.569414:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204957.569418:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204957.569423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443124224:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204957.569447:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443124352:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204957.569466:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204957.569475:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204957/real 1788204957] req@ffff90234519fb80 x1875064443124352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204957.569482:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443124352/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204957.569488:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204957.569490:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204957.569492:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443124352:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204957.569495:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443124480:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204957.569506:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204957.569512:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204957/real 1788204957] req@ffff90234519c700 x1875064443124480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204957.569518:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443124480/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204957.569529:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204957.569532:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204957.569535:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443124480:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204957.569540:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443124608:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204957.569558:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204957.569568:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204957/real 1788204957] req@ffff90234519c000 x1875064443124608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204957.569579:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443124608/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204957.569584:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204957.569586:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204957.569588:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443124608:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204958.401049:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204959.425010:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204960.449061:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204961.473077:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204962.497069:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204962.689007:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204962.689020:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204962.689024:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204962.689036:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4234 00000100:00080000:3.0:1788204962.689049:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204962.689058:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788204962.689107:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443124736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204962.689138:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204962.689142:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204962.689143:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204962.689147:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4234 00000100:00080000:3.0:1788204962.689150:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204962.689154:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204962.689166:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204962.689178:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064443124864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204962.689188:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443124736:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788204962.689191:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204962.689197:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204962.689199:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204962.689203:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4234 00000100:00080000:3.0:1788204962.689206:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204962.689210:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204962.689215:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204962.689228:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443124992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204962.689236:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204962.689240:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204962.689242:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204962.689250:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4234 00000100:00080000:3.0:1788204962.689254:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204962.689271:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788204962.689276:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788204962.689290:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788204962.689290:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443125120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204962.689341:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204962/real 1788204962] req@ffff90235268aa00 x1875064443124736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204962.689367:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443124736/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204962.689378:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204962.689382:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204962.689386:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443124736:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204962.689410:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443124864:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204962.689433:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204962.689445:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204962/real 1788204962] req@ffff902352688a80 x1875064443124864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204962.689456:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064443124864/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204962.689464:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204962.689466:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204962.689469:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443124864:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204962.689474:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443124992:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204962.689492:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204962.689502:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204962/real 1788204962] req@ffff90235268ad80 x1875064443124992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204962.689512:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443124992/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204962.689519:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204962.689523:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204962.689526:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443124992:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204962.689532:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443125120:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204962.689561:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204962.689573:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204962/real 1788204962] req@ffff90235268b480 x1875064443125120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204962.689584:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443125120/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204962.689592:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204962.689601:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204962.689604:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443125120:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204963.521078:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204964.545044:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204965.569027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204966.593042:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204967.617060:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204967.808956:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204967.808966:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204967.808970:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204967.808980:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4239 00000100:00080000:1.0:1788204967.808990:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204967.808997:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204967.809033:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443125248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204967.809058:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204967.809061:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204967.809062:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204967.809065:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4239 00000100:00080000:1.0:1788204967.809069:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204967.809073:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204967.809080:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204967.809091:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064443125376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204967.809099:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204967.809106:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788204967.809108:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443125248:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204967.809108:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204967.809123:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4239 00000100:00080000:1.0:1788204967.809126:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204967.809128:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204967.809131:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204967.809140:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443125504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204967.809146:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204967.809149:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204967.809150:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204967.809152:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4239 00000100:00080000:1.0:1788204967.809154:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204967.809156:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204967.809159:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204967.809165:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443125632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204967.809254:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443125376:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204967.809285:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443125504:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204967.809323:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443125632:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204968.641192:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204969.665040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204970.689061:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788204970.881234:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788204970.881290:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788204970.881311:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788204970.881365:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788204970.881377:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788204970.881386:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788204970.881486:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204967/real 1788204970] req@ffff90234519dc00 x1875064443125376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204970.881516:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064443125376/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204970.881531:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204970.881537:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204970.881542:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443125376:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204970.881551:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204967/real 1788204970] req@ffff90234519f100 x1875064443125504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204970.881571:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443125504/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204970.881581:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204970.881585:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204970.881588:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443125504:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204970.881596:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204967/real 1788204970] req@ffff90234519c380 x1875064443125632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204970.881609:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443125632/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204970.881617:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204970.881620:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204970.881624:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443125632:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204970.881631:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204967/real 1788204970] req@ffff90234519ea00 x1875064443125248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204970.881646:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443125248/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204970.881654:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204970.881657:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204970.881661:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443125248:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788204971.713008:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204972.736983:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204972.928952:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204972.928960:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204972.928963:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204972.928971:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4244 00000100:00080000:1.0:1788204972.928978:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204972.928983:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204972.929012:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443125760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204972.929031:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204972.929034:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204972.929035:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204972.929038:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4244 00000100:00080000:1.0:1788204972.929040:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204972.929042:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204972.929048:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204972.929056:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443125888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204972.929064:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204972.929066:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204972.929067:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204972.929069:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4244 00000100:00080000:1.0:1788204972.929071:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204972.929073:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204972.929076:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204972.929093:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443125760:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204972.929100:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443126016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204972.929111:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204972.929114:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204972.929115:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204972.929117:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4244 00000100:00080000:1.0:1788204972.929120:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204972.929123:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204972.929127:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204972.929138:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443126144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204972.929163:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204972.929198:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204972/real 1788204972] req@ffff90234519f480 x1875064443125760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204972.929212:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443125760/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204972.929219:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204972.929223:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204972.929227:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443125760:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204972.929234:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443125888:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204972.929257:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204972.929268:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204972/real 1788204972] req@ffff90234519ea00 x1875064443125888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204972.929278:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443125888/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204972.929285:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204972.929288:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204972.929291:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443125888:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204972.929314:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443126016:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204972.929333:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204972.929342:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204972/real 1788204972] req@ffff90234519c380 x1875064443126016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204972.929356:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443126016/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204972.929364:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204972.929366:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204972.929369:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443126016:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204972.929374:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443126144:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204972.929388:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204972.929397:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204972/real 1788204972] req@ffff90234519f100 x1875064443126144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204972.929406:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443126144/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204972.929412:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204972.929415:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204972.929417:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443126144:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204973.761004:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204974.785012:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204975.809091:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204976.833050:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204977.857038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204978.048981:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204978.048999:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204978.049003:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204978.049013:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4249 00000100:00080000:1.0:1788204978.049022:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204978.049029:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204978.049063:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064443126272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204978.049085:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204978.049088:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204978.049090:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204978.049101:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4249 00000100:00080000:1.0:1788204978.049105:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204978.049110:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204978.049118:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204978.049129:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443126272:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204978.049134:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443126400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204978.049147:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204978.049151:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204978.049152:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204978.049156:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4249 00000100:00080000:1.0:1788204978.049160:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204978.049164:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204978.049169:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204978.049181:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443126528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204978.049187:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788204978.049197:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204978.049200:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204978.049203:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204978.049207:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4249 00000100:00080000:1.0:1788204978.049212:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204978.049216:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788204978.049222:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204978/real 1788204978] req@ffff90234519dc00 x1875064443126272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788204978.049222:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204978.049240:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443126656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204978.049241:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064443126272/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204978.049267:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204978.049272:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204978.049277:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443126272:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204978.049299:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443126400:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204978.049320:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204978.049329:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204978/real 1788204978] req@ffff90234519c000 x1875064443126400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204978.049337:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443126400/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204978.049342:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204978.049344:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204978.049346:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443126400:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204978.049350:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443126528:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204978.049361:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204978.049366:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204978/real 1788204978] req@ffff90234519c700 x1875064443126528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204978.049373:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443126528/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204978.049377:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204978.049379:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204978.049381:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443126528:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204978.049384:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443126656:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204978.049392:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204978.049397:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204978/real 1788204978] req@ffff90234519fb80 x1875064443126656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204978.049403:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443126656/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204978.049408:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204978.049409:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204978.049411:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443126656:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788204978.881036:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204979.905054:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204980.929011:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204981.953025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204982.977078:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204983.169005:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204983.169011:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204983.169014:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204983.169022:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4254 00000100:00080000:2.0:1788204983.169032:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204983.169039:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204983.169081:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443126784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204983.169107:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204983.169111:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204983.169113:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204983.169117:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4254 00000100:00080000:2.0:1788204983.169121:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204983.169124:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204983.169133:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204983.169149:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443126912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204983.169151:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443126784:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204983.169166:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204983.169169:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204983.169171:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204983.169174:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4254 00000100:00080000:2.0:1788204983.169178:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204983.169181:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204983.169185:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204983.169196:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443127040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204983.169202:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788204983.169206:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788204983.169207:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204983.169209:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204983.169213:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4254 00000100:00080000:2.0:1788204983.169216:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204983.169218:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204983.169222:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204983.169229:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443127168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204983.169242:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204983/real 1788204983] req@ffff90234598e300 x1875064443126784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204983.169255:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443126784/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204983.169262:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204983.169265:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204983.169267:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443126784:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204983.169286:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443126912:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204983.169305:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204983.169317:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204983/real 1788204983] req@ffff90234598d180 x1875064443126912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204983.169328:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443126912/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204983.169335:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204983.169337:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204983.169339:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443126912:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204983.169344:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443127040:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204983.169370:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204983.169381:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204983/real 1788204983] req@ffff90234598c380 x1875064443127040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204983.169391:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443127040/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204983.169396:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204983.169398:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204983.169400:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443127040:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204983.169403:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443127168:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204983.169415:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204983.169421:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204983/real 1788204983] req@ffff90234598c700 x1875064443127168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204983.169431:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443127168/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204983.169439:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204983.169442:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204983.169444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443127168:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204984.001058:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204985.025032:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204986.049053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204987.073019:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204988.097089:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204988.289077:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204988.289088:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204988.289091:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204988.289104:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4259 00000100:00080000:2.0:1788204988.289114:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204988.289121:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204988.289158:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443127296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204988.289183:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204988.289187:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204988.289188:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204988.289193:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4259 00000100:00080000:2.0:1788204988.289198:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204988.289205:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204988.289215:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204988.289232:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443127296:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788204988.289233:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443127424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204988.289262:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204988.289266:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204988.289269:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204988.289274:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4259 00000100:00080000:2.0:1788204988.289279:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204988.289284:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204988.289289:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204988.289299:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443127552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204988.289307:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204988.289310:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204988.289312:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204988.289315:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4259 00000100:00080000:2.0:1788204988.289317:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204988.289319:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204988.289321:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204988.289326:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443127680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204988.289340:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204988.289390:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204988/real 1788204988] req@ffff90234598f480 x1875064443127296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204988.289409:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443127296/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204988.289420:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204988.289425:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204988.289429:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443127296:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788204988.289456:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443127424:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788204988.289475:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204988.289484:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204988/real 1788204988] req@ffff90234598ce00 x1875064443127424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204988.289496:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443127424/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204988.289510:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204988.289514:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204988.289517:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443127424:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788204988.289523:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443127552:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204988.289542:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204988.289552:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204988/real 1788204988] req@ffff90234598f800 x1875064443127552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204988.289562:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443127552/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204988.289570:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204988.289573:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204988.289576:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443127552:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788204988.289581:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443127680:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788204988.289600:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788204988.289611:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204988/real 1788204988] req@ffff90234598dc00 x1875064443127680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204988.289622:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443127680/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204988.289631:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204988.289634:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788204988.289639:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443127680:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204989.121062:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204990.145090:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204991.169014:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204992.193075:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204993.217043:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204993.409002:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204993.409013:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204993.409017:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204993.409027:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4264 00000100:00080000:1.0:1788204993.409038:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204993.409053:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788204993.409096:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443127808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204993.409123:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204993.409126:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204993.409127:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204993.409130:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4264 00000100:00080000:1.0:1788204993.409133:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204993.409135:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204993.409144:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204993.409154:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064443127936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204993.409162:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204993.409164:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204993.409166:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204993.409171:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4264 00000100:00080000:1.0:1788204993.409175:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788204993.409178:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443127808:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788204993.409180:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204993.409185:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204993.409196:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443128064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204993.409202:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204993.409205:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204993.409206:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204993.409209:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4264 00000100:00080000:1.0:1788204993.409211:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204993.409214:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788204993.409217:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204993.409223:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443128192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204993.409254:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204993.409287:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204993/real 1788204993] req@ffff90234519df80 x1875064443127808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204993.409299:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443127808/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204993.409306:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204993.409309:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204993.409312:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443127808:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204993.409330:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443127936:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204993.409343:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204993.409351:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204993/real 1788204993] req@ffff90234519f800 x1875064443127936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204993.409365:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064443127936/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204993.409373:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204993.409375:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204993.409377:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443127936:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204993.409382:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443128064:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204993.409393:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204993.409399:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204993/real 1788204993] req@ffff90234519ed80 x1875064443128064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204993.409406:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443128064/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204993.409414:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204993.409417:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204993.409420:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443128064:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204993.409425:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443128192:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204993.409443:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204993.409454:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204993/real 1788204993] req@ffff90234519d880 x1875064443128192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204993.409464:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443128192/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204993.409472:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204993.409475:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204993.409477:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443128192:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788204994.241060:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788204995.265072:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788204996.289018:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204997.313068:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204998.337084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204998.529008:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204998.529017:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204998.529020:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204998.529030:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4270 00000100:00080000:2.0:1788204998.529040:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204998.529047:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788204998.529079:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064443128320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204998.529101:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204998.529105:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204998.529107:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204998.529110:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4270 00000100:00080000:2.0:1788204998.529114:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204998.529117:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204998.529126:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204998.529137:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064443128448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204998.529145:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443128320:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788204998.529149:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204998.529154:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204998.529155:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204998.529160:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4270 00000100:00080000:2.0:1788204998.529163:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204998.529166:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204998.529171:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204998.529182:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064443128576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204998.529190:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204998.529193:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204998.529195:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204998.529198:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4270 00000800:00000100:0.0:1788204998.529203:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788204998.529203:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204998.529209:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788204998.529214:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204998.529226:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443128704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204998.529249:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204998/real 1788204998] req@ffff90234598d880 x1875064443128320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204998.529268:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064443128320/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204998.529277:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204998.529281:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204998.529285:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443128320:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788204998.529306:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443128448:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788204998.529326:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204998.529335:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204998/real 1788204998] req@ffff90234598e680 x1875064443128448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204998.529343:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064443128448/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204998.529348:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204998.529350:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204998.529352:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443128448:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788204998.529355:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443128576:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204998.529366:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204998.529371:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204998/real 1788204998] req@ffff90234598ca80 x1875064443128576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204998.529377:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064443128576/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204998.529385:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204998.529387:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204998.529389:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443128576:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788204998.529392:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443128704:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788204998.529400:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788204998.529405:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204998/real 1788204998] req@ffff90234598dc00 x1875064443128704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204998.529410:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443128704/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204998.529415:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204998.529417:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788204998.529418:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443128704:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788204999.361115:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788205000.385011:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205001.409035:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205002.433117:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205003.457049:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205003.649004:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205003.649013:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205003.649017:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205003.649026:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4275 00000100:00080000:3.0:1788205003.649037:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205003.649044:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788205003.649078:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443128832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205003.649101:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205003.649104:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205003.649105:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205003.649109:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4275 00000100:00080000:3.0:1788205003.649112:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205003.649116:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205003.649124:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205003.649137:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443128960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205003.649146:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788205003.649149:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443128832:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788205003.649150:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205003.649154:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205003.649162:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4275 00000100:00080000:3.0:1788205003.649164:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205003.649168:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205003.649172:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205003.649182:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443129088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205003.649195:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205003.649198:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205003.649199:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205003.649203:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4275 00000100:00080000:3.0:1788205003.649207:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205003.649211:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205003.649215:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788205003.649216:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205003.649228:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064443129216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205003.649258:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205003/real 1788205003] req@ffff902352688380 x1875064443128832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205003.649272:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443128832/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205003.649279:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205003.649282:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205003.649285:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443128832:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205003.649302:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443128960:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205003.649316:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205003.649324:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205003/real 1788205003] req@ffff90235268b480 x1875064443128960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205003.649338:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443128960/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205003.649346:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205003.649348:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205003.649351:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443128960:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205003.649354:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443129088:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205003.649365:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205003.649372:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205003/real 1788205003] req@ffff90235268ad80 x1875064443129088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205003.649379:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443129088/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205003.649384:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205003.649386:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205003.649388:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443129088:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205003.649390:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443129216:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205003.649399:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205003.649404:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205003/real 1788205003] req@ffff902352688a80 x1875064443129216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205003.649411:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064443129216/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205003.649416:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205003.649417:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205003.649419:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443129216:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788205004.480992:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205005.505034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205006.529014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205007.553025:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205008.576980:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205008.768985:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205008.768992:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205008.768995:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205008.769002:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4280 00000100:00080000:3.0:1788205008.769010:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205008.769015:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788205008.769044:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443129344/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205008.769065:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205008.769067:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205008.769069:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205008.769071:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4280 00000100:00080000:3.0:1788205008.769073:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205008.769075:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205008.769082:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205008.769089:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443129472/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205008.769094:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205008.769096:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205008.769097:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205008.769099:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4280 00000100:00080000:3.0:1788205008.769100:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205008.769102:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205008.769104:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205008.769108:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443129600/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205008.769113:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205008.769115:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205008.769115:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205008.769117:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4280 00000100:00080000:3.0:1788205008.769119:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205008.769120:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205008.769123:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205008.769127:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443129728/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205008.769153:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443129344:192.168.201.130@tcp:250:kworker.0 00000800:00000100:0.0:1788205008.769201:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205008.769232:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205008/real 1788205008] req@ffff90235268aa00 x1875064443129344/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205008.769251:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443129344/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205008.769261:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205008.769264:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205008.769266:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443129344:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205008.769271:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443129472:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205008.769287:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205008.769296:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205008/real 1788205008] req@ffff902352689880 x1875064443129472/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205008.769303:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443129472/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205008.769308:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205008.769310:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205008.769312:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443129472:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205008.769315:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443129600:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205008.769325:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205008.769330:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205008/real 1788205008] req@ffff90235268bb80 x1875064443129600/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205008.769336:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443129600/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205008.769341:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205008.769343:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205008.769344:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443129600:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205008.769347:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443129728:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205008.769356:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205008.769361:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205008/real 1788205008] req@ffff902352688700 x1875064443129728/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205008.769367:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443129728/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205008.769372:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205008.769373:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205008.769375:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443129728:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205009.601044:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205010.625005:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205011.649071:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205012.673078:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205013.697036:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205013.888969:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205013.888979:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205013.888982:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205013.888991:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4285 00000100:00080000:1.0:1788205013.889000:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205013.889006:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788205013.889035:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443129856/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205013.889057:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205013.889064:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205013.889066:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205013.889070:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4285 00000100:00080000:1.0:1788205013.889074:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205013.889079:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205013.889087:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205013.889098:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443129856:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788205013.889104:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443129984/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205013.889125:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205013.889128:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788205013.889130:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205013.889134:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4285 00000100:00080000:1.0:1788205013.889138:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205013.889141:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205013.889145:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788205013.889149:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205013.889161:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443130112/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205013.889170:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205013.889174:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788205013.889175:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205013.889178:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4285 00000100:00080000:1.0:1788205013.889181:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205013.889184:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205013.889189:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205013.889192:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205013/real 1788205013] req@ffff90234519e300 x1875064443129856/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205013.889202:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064443130240/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205013.889219:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443129856/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205013.889231:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205013.889236:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205013.889241:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443129856:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205013.889266:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443129984:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205013.889289:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205013.889301:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205013/real 1788205013] req@ffff90234519d880 x1875064443129984/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205013.889313:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443129984/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205013.889321:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205013.889323:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205013.889325:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443129984:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205013.889329:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443130112:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205013.889346:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205013.889356:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205013/real 1788205013] req@ffff90234519ed80 x1875064443130112/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205013.889367:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443130112/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205013.889399:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205013.889402:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205013.889406:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443130112:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205013.889412:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443130240:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205013.889431:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205013.889442:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205013/real 1788205013] req@ffff90234519f800 x1875064443130240/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205013.889451:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064443130240/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205013.889456:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205013.889458:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205013.889460:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443130240:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205014.721045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205015.745039:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205016.769086:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205017.793065:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205018.817027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205019.009041:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205019.009054:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205019.009059:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205019.009072:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4290 00000100:00080000:2.0:1788205019.009087:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205019.009095:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205019.009144:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443130368/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205019.009183:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205019.009188:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205019.009190:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205019.009194:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4290 00000100:00080000:2.0:1788205019.009198:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205019.009202:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205019.009214:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205019.009227:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443130496/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205019.009238:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443130368:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788205019.009241:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205019.009248:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205019.009250:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205019.009255:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4290 00000100:00080000:2.0:1788205019.009259:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205019.009263:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205019.009268:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205019.009295:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443130624/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205019.009306:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205019.009310:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205019.009312:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205019.009317:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4290 00000100:00080000:2.0:1788205019.009323:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205019.009327:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205019.009333:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788205019.009335:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788205019.009350:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443130752/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205019.009381:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205019/real 1788205019] req@ffff90234598f800 x1875064443130368/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205019.009401:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443130368/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205019.009409:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205019.009412:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205019.009415:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443130368:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205019.009436:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443130496:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205019.009451:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205019.009466:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205019/real 1788205019] req@ffff90234598ce00 x1875064443130496/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205019.009479:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443130496/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205019.009487:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205019.009490:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205019.009494:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443130496:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205019.009500:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443130624:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205019.009526:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205019.009534:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205019/real 1788205019] req@ffff90234598f480 x1875064443130624/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205019.009541:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443130624/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205019.009551:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205019.009553:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205019.009555:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443130624:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205019.009558:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443130752:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205019.009568:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205019.009573:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205019/real 1788205019] req@ffff90234598c700 x1875064443130752/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205019.009580:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443130752/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205019.009585:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205019.009586:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205019.009588:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443130752:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788205019.841086:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205020.865024:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205021.889276:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205022.913034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205023.937047:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205024.129026:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205024.129039:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205024.129043:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205024.129056:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4295 00000100:00080000:2.0:1788205024.129069:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205024.129078:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205024.129126:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443130880/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205024.129162:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205024.129166:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205024.129168:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205024.129178:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4295 00000100:00080000:2.0:1788205024.129182:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205024.129187:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205024.129209:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205024.129226:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443131008/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205024.129227:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443130880:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788205024.129244:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205024.129247:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205024.129249:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205024.129253:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4295 00000100:00080000:2.0:1788205024.129257:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205024.129262:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205024.129267:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205024.129279:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443131136/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205024.129288:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205024.129292:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205024.129293:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205024.129297:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4295 00000100:00080000:2.0:1788205024.129301:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788205024.129303:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788205024.129308:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205024.129313:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205024.129325:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064443131264/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205024.129351:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205024/real 1788205024] req@ffff90234598c380 x1875064443130880/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205024.129366:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443130880/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205024.129375:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205024.129378:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205024.129381:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443130880:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205024.129404:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443131008:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205024.129420:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205024.129430:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205024/real 1788205024] req@ffff90234598d180 x1875064443131008/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205024.129439:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443131008/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205024.129444:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205024.129446:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205024.129448:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443131008:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205024.129453:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443131136:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205024.129469:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205024.129476:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205024/real 1788205024] req@ffff90234598e300 x1875064443131136/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205024.129482:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443131136/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205024.129488:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205024.129489:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205024.129492:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443131136:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205024.129496:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443131264:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205024.129511:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205024.129516:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205024/real 1788205024] req@ffff90234598f100 x1875064443131264/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205024.129524:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064443131264/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205024.129529:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205024.129531:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205024.129533:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443131264:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205024.961032:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205025.985031:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205027.009062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205028.033031:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205029.057021:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205029.248973:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205029.248981:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205029.248984:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205029.248993:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4300 00000100:00080000:2.0:1788205029.249003:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205029.249010:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205029.249045:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064443131392/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205029.249068:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205029.249070:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205029.249071:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205029.249074:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4300 00000100:00080000:2.0:1788205029.249077:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205029.249079:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205029.249085:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205029.249093:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064443131520/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205029.249099:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205029.249102:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205029.249102:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205029.249126:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4300 00000100:00100000:0.0:1788205029.249128:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443131392:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788205029.249130:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205029.249136:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205029.249141:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205029.249152:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443131648/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205029.249159:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205029.249162:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205029.249163:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205029.249166:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4300 00000100:00080000:2.0:1788205029.249168:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205029.249171:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205029.249174:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205029.249181:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064443131776/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205029.249185:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205029.249220:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205029/real 1788205029] req@ffff90234598ed80 x1875064443131392/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205029.249232:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064443131392/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205029.249240:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205029.249242:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205029.249245:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443131392:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205029.249249:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443131520:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205029.249262:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205029.249269:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205029/real 1788205029] req@ffff90234598fb80 x1875064443131520/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205029.249275:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064443131520/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205029.249284:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205029.249285:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205029.249287:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443131520:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205029.249305:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443131648:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205029.249314:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205029.249320:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205029/real 1788205029] req@ffff90234598df80 x1875064443131648/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205029.249326:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443131648/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205029.249331:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205029.249332:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205029.249334:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443131648:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205029.249337:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443131776:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205029.249345:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205029.249350:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205029/real 1788205029] req@ffff90234598f100 x1875064443131776/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205029.249356:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064443131776/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205029.249360:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205029.249362:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205029.249364:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443131776:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205030.080995:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205031.105037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205032.129005:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205033.153034:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205034.177118:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205034.368998:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205034.369008:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205034.369012:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205034.369034:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4305 00000100:00080000:2.0:1788205034.369052:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205034.369059:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205034.369094:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443131904/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205034.369125:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205034.369129:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205034.369131:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205034.369135:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4305 00000100:00080000:2.0:1788205034.369138:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205034.369142:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205034.369151:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205034.369165:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443132032/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205034.369170:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443131904:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788205034.369183:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205034.369186:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205034.369188:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205034.369193:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4305 00000100:00080000:2.0:1788205034.369197:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205034.369201:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205034.369207:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205034.369220:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443132160/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205034.369229:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205034.369232:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205034.369234:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205034.369238:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4305 00000100:00080000:2.0:1788205034.369241:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205034.369246:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205034.369252:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205034.369265:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443132032:192.168.201.130@tcp:38:kworker.0 00000100:00100000:2.0:1788205034.369266:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443132288/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205034.369300:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443132160:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205034.369318:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443132288:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205035.201126:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205036.225106:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205037.249035:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788205037.441190:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:1.0:1788205037.441235:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788205037.441251:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788205037.441296:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788205037.441307:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:1.0:1788205037.441315:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788205037.441403:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205034/real 1788205037] req@ffff90234598c700 x1875064443132288/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205037.441430:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443132288/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205037.441443:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205037.441449:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205037.441460:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443132288:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205037.441469:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205034/real 1788205037] req@ffff90234598d180 x1875064443132032/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205037.441482:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443132032/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205037.441491:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205037.441495:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205037.441498:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443132032:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205037.441505:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205034/real 1788205037] req@ffff90234598c380 x1875064443132160/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205037.441522:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443132160/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205037.441531:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205037.441535:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205037.441538:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443132160:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205037.441545:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205034/real 1788205037] req@ffff90234598e300 x1875064443131904/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205037.441559:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443131904/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205037.441567:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205037.441571:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205037.441574:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443131904:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788205038.273111:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205039.297033:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205039.488975:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205039.488985:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205039.488989:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205039.488999:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4311 00000100:00080000:2.0:1788205039.489008:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205039.489014:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205039.489053:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443132416/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205039.489077:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205039.489080:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205039.489081:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205039.489084:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4311 00000100:00080000:2.0:1788205039.489087:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205039.489096:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205039.489105:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205039.489119:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443132544/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205039.489129:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205039.489134:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788205039.489136:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443132416:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788205039.489137:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205039.489144:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4311 00000100:00080000:2.0:1788205039.489149:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205039.489152:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205039.489157:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205039.489169:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443132672/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205039.489184:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205039.489187:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205039.489189:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205039.489192:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4311 00000100:00080000:2.0:1788205039.489195:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205039.489199:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205039.489204:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205039.489216:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443132800/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205039.489219:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205039.489461:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205039/real 1788205039] req@ffff90234598f480 x1875064443132416/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205039.489480:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443132416/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205039.489489:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205039.489493:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205039.489497:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443132416:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205039.489520:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443132544:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205039.489543:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205039.489554:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205039/real 1788205039] req@ffff90234598ce00 x1875064443132544/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205039.489565:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443132544/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205039.489573:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205039.489576:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205039.489579:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443132544:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205039.489585:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443132672:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205039.489602:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205039.489615:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205039/real 1788205039] req@ffff90234598f800 x1875064443132672/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205039.489627:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443132672/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205039.489634:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205039.489637:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205039.489640:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443132672:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205039.489646:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443132800:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205039.489665:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205039.489675:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205039/real 1788205039] req@ffff90234598dc00 x1875064443132800/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205039.489687:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443132800/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205039.489695:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205039.489697:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205039.489700:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443132800:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788205040.321108:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205041.345095:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205042.369068:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205043.393035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205044.416981:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205044.608982:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205044.608989:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205044.608992:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205044.609000:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4316 00000100:00080000:1.0:1788205044.609007:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205044.609012:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788205044.609042:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443132928/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205044.609061:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205044.609063:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205044.609064:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205044.609067:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4316 00000100:00080000:1.0:1788205044.609069:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205044.609071:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205044.609077:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205044.609085:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443133056/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205044.609093:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205044.609095:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788205044.609096:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788205044.609099:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443132928:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788205044.609101:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4316 00000100:00080000:1.0:1788205044.609106:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205044.609109:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205044.609112:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205044.609119:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443133184/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205044.609125:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205044.609127:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788205044.609128:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205044.609130:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4316 00000100:00080000:1.0:1788205044.609140:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205044.609143:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205044.609145:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788205044.609151:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205044.609153:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443133312/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205044.609183:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205044/real 1788205044] req@ffff90234519df80 x1875064443132928/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205044.609196:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443132928/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205044.609203:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205044.609205:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205044.609208:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443132928:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205044.609223:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443133056:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205044.609243:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205044.609253:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205044/real 1788205044] req@ffff90234519fb80 x1875064443133056/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205044.609260:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443133056/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205044.609268:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205044.609270:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205044.609272:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443133056:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205044.609275:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443133184:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205044.609285:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205044.609290:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205044/real 1788205044] req@ffff90234519c700 x1875064443133184/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205044.609296:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443133184/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205044.609301:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205044.609302:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205044.609304:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443133184:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205044.609307:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443133312:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205044.609315:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205044.609320:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205044/real 1788205044] req@ffff90234519c000 x1875064443133312/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205044.609326:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443133312/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205044.609331:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205044.609333:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205044.609337:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443133312:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205045.440974:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205046.465026:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205047.489049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205048.513051:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205049.537037:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205049.728990:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205049.728997:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205049.729000:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205049.729007:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4321 00000100:00080000:2.0:1788205049.729016:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205049.729021:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205049.729054:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064443133440/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205049.729089:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205049.729093:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205049.729095:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205049.729100:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4321 00000100:00080000:2.0:1788205049.729103:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205049.729107:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205049.729114:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205049.729123:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064443133568/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205049.729131:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205049.729133:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205049.729135:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788205049.729138:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443133440:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788205049.729139:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4321 00000100:00080000:2.0:1788205049.729143:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205049.729148:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205049.729151:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205049.729157:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064443133696/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205049.729163:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205049.729165:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205049.729166:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205049.729168:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4321 00000100:00080000:2.0:1788205049.729170:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205049.729172:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205049.729174:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205049.729181:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443133824/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205049.729201:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205049.729243:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205049/real 1788205049] req@ffff90234598ca80 x1875064443133440/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205049.729260:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064443133440/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205049.729270:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205049.729274:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205049.729277:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443133440:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205049.729299:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443133568:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205049.729319:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205049.729330:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205049/real 1788205049] req@ffff90234598e680 x1875064443133568/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205049.729342:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064443133568/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205049.729349:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205049.729353:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205049.729356:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443133568:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205049.729362:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443133696:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205049.729378:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205049.729387:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205049/real 1788205049] req@ffff90234598d880 x1875064443133696/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205049.729399:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064443133696/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205049.729407:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205049.729410:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205049.729412:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443133696:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205049.729418:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443133824:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205049.729433:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205049.729442:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205049/real 1788205049] req@ffff90234598dc00 x1875064443133824/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205049.729453:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443133824/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205049.729460:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205049.729463:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205049.729466:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443133824:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205050.561057:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205051.584995:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205052.609242:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205053.633046:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205054.657040:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205054.848991:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205054.849178:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205054.849184:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205054.849196:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4326 00000100:00080000:2.0:1788205054.849207:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205054.849215:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205054.849253:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443133952/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205054.849364:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205054.849369:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205054.849371:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205054.849376:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4326 00000100:00080000:2.0:1788205054.849380:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205054.849385:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205054.849395:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205054.849420:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443134080/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205054.849430:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205054.849433:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205054.849435:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205054.849439:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4326 00000100:00080000:2.0:1788205054.849443:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205054.849449:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205054.849455:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205054.849469:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443133952:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788205054.849470:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443134208/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205054.849486:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205054.849490:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205054.849492:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205054.849496:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4326 00000100:00080000:2.0:1788205054.849499:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205054.849504:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205054.849509:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205054.849520:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443134336/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205054.850808:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205054.850867:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205054/real 1788205054] req@ffff90234598f800 x1875064443133952/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205054.850897:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443133952/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205054.850908:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205054.850912:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205054.850916:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443133952:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205054.850925:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443134080:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205054.850949:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205054.850959:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205054/real 1788205054] req@ffff90234598ce00 x1875064443134080/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205054.850967:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443134080/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205054.850973:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205054.850976:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205054.850979:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443134080:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205054.851017:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443134208:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205054.851039:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205054.851050:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205054/real 1788205054] req@ffff90234598f480 x1875064443134208/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205054.851061:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443134208/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205054.851069:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205054.851072:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205054.851075:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443134208:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205054.851082:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443134336:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205054.851097:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205054.851106:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205054/real 1788205054] req@ffff90234598e300 x1875064443134336/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205054.851116:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443134336/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205054.851123:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205054.851126:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205054.851129:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443134336:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205055.681043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205056.705067:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205057.729051:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205058.753044:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205059.777896:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205059.969018:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205059.969029:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205059.969033:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205059.969043:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4331 00000100:00080000:2.0:1788205059.969054:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205059.969061:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205059.969100:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443134464/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205059.969127:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205059.969131:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205059.969133:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205059.969137:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4331 00000100:00080000:2.0:1788205059.969141:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205059.969144:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205059.969153:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205059.969165:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443134592/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205059.969175:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205059.969178:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205059.969180:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205059.969185:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4331 00000100:00080000:2.0:1788205059.969189:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205059.969193:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205059.969197:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205059.969208:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443134720/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205059.969218:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788205059.969228:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443134464:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788205059.969229:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205059.969234:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205059.969245:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4331 00000100:00080000:2.0:1788205059.969249:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205059.969252:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205059.969257:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205059.969270:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064443134848/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205059.969335:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205059.969400:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205059/real 1788205059] req@ffff90234598c380 x1875064443134464/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205059.969423:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443134464/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205059.969435:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205059.969440:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205059.969444:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443134464:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205059.969452:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443134592:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205059.969475:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205059.969489:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205059/real 1788205059] req@ffff90234598d180 x1875064443134592/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205059.969502:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443134592/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205059.969510:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205059.969513:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205059.969517:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443134592:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205059.969547:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443134720:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205059.969568:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205059.969595:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205059/real 1788205059] req@ffff90234598c700 x1875064443134720/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205059.969607:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443134720/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205059.969615:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205059.969619:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205059.969622:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443134720:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205059.969627:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443134848:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205059.969645:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205059.969654:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205059/real 1788205059] req@ffff90234598f100 x1875064443134848/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205059.969665:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064443134848/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205059.969678:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205059.969681:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205059.969684:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443134848:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788205060.801591:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205061.825224:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205062.849126:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205063.873067:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205064.897044:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205065.089032:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205065.089044:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205065.089048:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205065.089061:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4336 00000100:00080000:2.0:1788205065.089074:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205065.089082:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205065.089130:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443134976/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205065.089161:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205065.089166:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205065.089168:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205065.089172:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4336 00000100:00080000:2.0:1788205065.089177:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205065.089182:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205065.089194:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205065.089210:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443134976:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788205065.089223:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064443135104/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205065.089235:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205065.089239:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205065.089241:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205065.089246:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4336 00000100:00080000:2.0:1788205065.089250:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205065.089255:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205065.089261:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205065.089276:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064443135232/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205065.089285:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788205065.089286:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788205065.089291:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205065.089294:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205065.089299:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4336 00000100:00080000:2.0:1788205065.089302:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205065.089307:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205065.089312:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205065.089323:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064443135360/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205065.089339:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205065/real 1788205065] req@ffff90234598df80 x1875064443134976/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205065.089356:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443134976/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205065.089366:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205065.089371:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205065.089374:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443134976:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205065.089402:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443135104:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205065.089424:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205065.089436:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205065/real 1788205065] req@ffff90234598fb80 x1875064443135104/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205065.089448:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064443135104/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205065.089456:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205065.089459:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205065.089462:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443135104:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205065.089469:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443135232:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205065.089677:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205065.089696:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205065/real 1788205065] req@ffff90234598ed80 x1875064443135232/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205065.089707:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064443135232/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205065.089715:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205065.089718:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205065.089720:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443135232:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205065.089726:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443135360:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205065.089746:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205065.089755:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205065/real 1788205065] req@ffff90234598f100 x1875064443135360/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205065.089764:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064443135360/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205065.089770:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205065.089773:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205065.089776:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443135360:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205065.921089:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205066.945057:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205067.969020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205068.993169:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205070.017049:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205070.208935:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205070.208946:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205070.208950:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205070.208960:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4341 00000100:00080000:2.0:1788205070.208971:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205070.208978:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205070.209013:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443135488/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205070.209039:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205070.209042:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205070.209043:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205070.209046:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4341 00000100:00080000:2.0:1788205070.209048:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205070.209051:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205070.209057:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205070.209065:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443135616/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205070.209071:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205070.209073:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205070.209074:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205070.209076:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4341 00000100:00080000:2.0:1788205070.209078:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205070.209080:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205070.209083:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205070.209091:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443135488:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788205070.209091:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443135744/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205070.209105:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205070.209107:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205070.209108:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205070.209111:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4341 00000100:00080000:2.0:1788205070.209115:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205070.209119:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205070.209123:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205070.209134:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443135872/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205070.209166:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205070.209208:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205070/real 1788205070] req@ffff90234598c700 x1875064443135488/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205070.209227:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443135488/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205070.209238:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205070.209242:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205070.209245:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443135488:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205070.209252:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443135616:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205070.209270:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205070.209281:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205070/real 1788205070] req@ffff90234598d180 x1875064443135616/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205070.209292:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443135616/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205070.209300:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205070.209303:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205070.209307:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443135616:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205070.209327:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443135744:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205070.209344:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205070.209353:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205070/real 1788205070] req@ffff90234598c380 x1875064443135744/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205070.209365:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443135744/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205070.209379:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205070.209382:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205070.209385:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443135744:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205070.209390:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443135872:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205070.209424:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205070.209434:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205070/real 1788205070] req@ffff90234598e300 x1875064443135872/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205070.209444:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443135872/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205070.209451:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205070.209454:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205070.209457:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443135872:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205071.041034:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205072.065098:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205073.089201:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205074.113056:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205075.137066:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205075.329026:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205075.329037:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205075.329042:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205075.329053:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4346 00000100:00080000:0.0:1788205075.329064:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205075.329071:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788205075.329113:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443136000/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205075.329142:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205075.329146:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205075.329148:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205075.329152:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4346 00000100:00080000:0.0:1788205075.329160:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205075.329165:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205075.329176:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205075.329189:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443136128/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205075.329194:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443136000:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788205075.329199:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205075.329202:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205075.329203:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205075.329207:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4346 00000100:00080000:0.0:1788205075.329210:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205075.329213:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205075.329217:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205075.329225:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443136256/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205075.329233:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205075.329236:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205075.329238:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205075.329241:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4346 00000100:00080000:0.0:1788205075.329244:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205075.329248:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205075.329252:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205075.329259:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443136384/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205075.329291:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205075.329356:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205075/real 1788205075] req@ffff9023450baa00 x1875064443136000/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205075.329373:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443136000/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205075.329383:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205075.329386:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205075.329390:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443136000:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788205075.329411:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443136128:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788205075.329428:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205075.329437:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205075/real 1788205075] req@ffff9023450b9180 x1875064443136128/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205075.329444:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443136128/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205075.329450:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205075.329452:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205075.329454:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443136128:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788205075.329457:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443136256:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205075.329467:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205075.329472:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205075/real 1788205075] req@ffff9023450b8000 x1875064443136256/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205075.329479:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443136256/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205075.329484:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205075.329485:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205075.329487:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443136256:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788205075.329490:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443136384:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205075.329498:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205075.329504:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205075/real 1788205075] req@ffff9023450ba300 x1875064443136384/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205075.329511:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443136384/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205075.329515:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205075.329517:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205075.329518:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443136384:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205076.161155:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205077.188510:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205078.209031:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205079.233267:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205080.257111:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205080.449002:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205080.449016:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205080.449021:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205080.449035:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4351 00000100:00080000:1.0:1788205080.449049:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205080.449057:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788205080.449103:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064443136512/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205080.449136:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205080.449141:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205080.449143:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205080.449147:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4351 00000100:00080000:1.0:1788205080.449152:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205080.449157:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205080.449170:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205080.449171:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443136512:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788205080.449189:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443136640/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205080.449198:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205080.449201:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788205080.449202:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205080.449205:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4351 00000100:00080000:1.0:1788205080.449208:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205080.449211:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205080.449216:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205080.449241:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443136768/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205080.449261:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788205080.449264:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205080.449269:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788205080.449270:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205080.449274:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4351 00000100:00080000:1.0:1788205080.449277:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205080.449281:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205080.449285:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205080.449294:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443136896/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205080.449321:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205080/real 1788205080] req@ffff90234519dc00 x1875064443136512/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205080.449342:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064443136512/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205080.449354:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205080.449358:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205080.449362:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443136512:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205080.449390:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443136640:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205080.449414:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205080.449427:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205080/real 1788205080] req@ffff90234519f100 x1875064443136640/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205080.449439:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443136640/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205080.449447:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205080.449450:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205080.449453:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443136640:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205080.449459:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443136768:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205080.449478:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205080.449486:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205080/real 1788205080] req@ffff90234519c380 x1875064443136768/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205080.449494:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443136768/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205080.449499:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205080.449501:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205080.449503:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443136768:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205080.449507:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443136896:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205080.449520:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205080.449525:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205080/real 1788205080] req@ffff90234519ea00 x1875064443136896/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205080.449533:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443136896/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205080.449545:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205080.449548:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205080.449551:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443136896:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788205081.281036:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205082.305015:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205083.329056:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205084.353056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205085.377060:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205085.568977:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205085.568987:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205085.568990:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205085.568998:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4357 00000100:00080000:1.0:1788205085.569008:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205085.569013:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788205085.569046:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443137024/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205085.569076:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205085.569079:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205085.569081:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205085.569084:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4357 00000100:00080000:1.0:1788205085.569086:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205085.569089:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205085.569096:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205085.569109:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443137152/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205085.569128:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205085.569147:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788205085.569149:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205085.569152:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4357 00000100:00080000:1.0:1788205085.569156:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205085.569161:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205085.569167:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205085.569171:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443137024:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788205085.569182:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443137280/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205085.569193:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205085.569196:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788205085.569197:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205085.569200:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4357 00000100:00080000:1.0:1788205085.569203:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205085.569205:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205085.569210:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205085.569221:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443137408/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205085.569240:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205085.569292:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205085/real 1788205085] req@ffff90234519f480 x1875064443137024/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205085.569315:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443137024/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205085.569324:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205085.569328:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205085.569331:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443137024:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205085.569338:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443137152:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205085.569367:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205085.569383:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205085/real 1788205085] req@ffff90234519ea00 x1875064443137152/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205085.569395:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443137152/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205085.569400:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205085.569402:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205085.569404:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443137152:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205085.569430:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443137280:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205085.569444:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205085.569451:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205085/real 1788205085] req@ffff90234519c380 x1875064443137280/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205085.569458:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443137280/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205085.569466:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205085.569468:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205085.569470:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443137280:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205085.569473:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443137408:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205085.569482:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205085.569490:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205085/real 1788205085] req@ffff90234519f100 x1875064443137408/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205085.569499:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443137408/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205085.569506:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205085.569508:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205085.569511:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443137408:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205086.401025:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205087.425005:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205088.448994:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205089.473158:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205090.497918:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205090.689501:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205090.689515:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205090.689520:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205090.689533:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4362 00000100:00080000:1.0:1788205090.689547:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205090.689555:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788205090.689603:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064443137536/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205090.689641:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205090.689646:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205090.689647:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205090.689652:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4362 00000100:00080000:1.0:1788205090.689655:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205090.689659:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205090.689670:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205090.689683:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443137664/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205090.689693:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205090.689697:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788205090.689699:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205090.689703:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4362 00000100:00080000:1.0:1788205090.689706:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205090.689712:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788205090.689713:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443137536:192.168.201.130@tcp:250:kworker.0 00000008:00100000:1.0:1788205090.689717:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205090.689729:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443137792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205090.689737:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205090.689741:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788205090.689743:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205090.689746:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4362 00000100:00080000:1.0:1788205090.689749:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205090.689752:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205090.689757:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205090.689766:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443137920/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205090.689799:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205090.689848:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205090/real 1788205090] req@ffff90234519dc00 x1875064443137536/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205090.689866:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064443137536/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205090.689898:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205090.689903:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205090.689907:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443137536:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205090.689915:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443137664:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205090.689938:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205090.689952:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205090/real 1788205090] req@ffff90234519c000 x1875064443137664/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205090.689964:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443137664/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205090.689971:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205090.689974:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205090.689978:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443137664:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205090.690004:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443137792:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205090.690023:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205090.690033:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205090/real 1788205090] req@ffff90234519c700 x1875064443137792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205090.690059:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443137792/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205090.690068:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205090.690071:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205090.690074:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443137792:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205090.690080:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443137920:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205090.690098:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205090.690108:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205090/real 1788205090] req@ffff90234519fb80 x1875064443137920/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205090.690120:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443137920/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205090.690127:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205090.690130:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205090.690133:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443137920:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205091.521267:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205092.545201:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205093.569064:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205094.593074:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205095.617044:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205095.809011:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205095.809022:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205095.809027:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205095.809038:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4367 00000100:00080000:3.0:1788205095.809048:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205095.809056:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788205095.809088:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443138048/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205095.809123:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205095.809128:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205095.809130:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205095.809135:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4367 00000100:00080000:3.0:1788205095.809139:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205095.809144:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205095.809154:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205095.809163:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443138048:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788205095.809171:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443138176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205095.809183:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205095.809186:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205095.809188:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205095.809193:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4367 00000100:00080000:3.0:1788205095.809197:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205095.809201:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205095.809206:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205095.809220:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443138304/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205095.809234:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788205095.809247:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205095.809249:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205095.809251:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205095.809255:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4367 00000100:00080000:3.0:1788205095.809258:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205095.809261:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205095.809265:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205095.809277:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205095/real 1788205095] req@ffff902352689180 x1875064443138048/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205095.809295:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443138048/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205095.809306:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205095.809310:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205095.809315:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443138048:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788205095.809334:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443138432/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205095.809342:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443138176:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788205095.809367:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205095.809384:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205095/real 1788205095] req@ffff902352689500 x1875064443138176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205095.809399:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443138176/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205095.809406:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205095.809410:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205095.809413:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443138176:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788205095.809418:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443138304:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205095.809435:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205095.809445:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205095/real 1788205095] req@ffff90235268a680 x1875064443138304/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205095.809455:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443138304/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205095.809463:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205095.809466:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205095.809470:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443138304:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788205095.809483:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443138432:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205095.809500:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205095.809511:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205095/real 1788205095] req@ffff902352688e00 x1875064443138432/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205095.809521:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443138432/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205095.809529:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205095.809532:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205095.809536:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443138432:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205096.641080:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205097.665073:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205098.689087:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205099.715296:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205100.737210:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205100.929057:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205100.929072:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205100.929077:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205100.929089:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4372 00000100:00080000:1.0:1788205100.929102:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205100.929111:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788205100.929161:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443138560/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205100.929198:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205100.929202:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205100.929208:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205100.929214:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4372 00000100:00080000:1.0:1788205100.929218:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205100.929222:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205100.929234:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205100.929249:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064443138688/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205100.929257:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443138560:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788205100.929263:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205100.929268:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788205100.929270:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205100.929275:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4372 00000100:00080000:1.0:1788205100.929284:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205100.929289:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205100.929294:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205100.929307:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443138816/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205100.929316:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205100.929320:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788205100.929322:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205100.929326:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4372 00000100:00080000:1.0:1788205100.929330:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205100.929334:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205100.929338:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205100.929348:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443138944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205100.929404:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443138688:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205100.929439:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443138816:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205100.929465:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443138944:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788205101.761111:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205102.785087:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205103.809400:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788205104.002070:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:3.0:1788205104.002113:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788205104.002129:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:3.0:1788205104.002169:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:3.0:1788205104.002178:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:3.0:1788205104.002186:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788205104.002452:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205100/real 1788205104] req@ffff90234519f800 x1875064443138688/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205104.002482:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064443138688/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205104.002494:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205104.002499:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205104.002503:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443138688:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205104.002511:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205100/real 1788205104] req@ffff90234519ed80 x1875064443138816/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205104.002522:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443138816/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205104.002531:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205104.002534:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205104.002537:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443138816:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205104.002543:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205100/real 1788205104] req@ffff90234519d880 x1875064443138944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205104.002554:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443138944/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205104.002561:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205104.002564:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205104.002567:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443138944:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205104.002574:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205100/real 1788205104] req@ffff90234519df80 x1875064443138560/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205104.002585:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443138560/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205104.002591:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205104.002594:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205104.002597:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443138560:192.168.201.130@tcp:250:kworker.0 00010000:00010000:1.0:1788205104.833167:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205105.857035:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205106.049958:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205106.049968:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205106.049972:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205106.049981:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4377 00000100:00080000:0.0:1788205106.049992:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205106.049999:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788205106.050033:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443139072/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205106.050079:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205106.050082:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205106.050084:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205106.050089:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4377 00000100:00080000:0.0:1788205106.050092:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205106.050096:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205106.050105:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205106.050119:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443139200/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205106.050128:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443139072:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788205106.050133:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205106.050137:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205106.050139:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205106.050143:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4377 00000100:00080000:0.0:1788205106.050152:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205106.050156:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205106.050161:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205106.050173:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443139328/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205106.050182:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205106.050186:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205106.050188:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788205106.050191:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788205106.050195:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4377 00000100:00080000:0.0:1788205106.050201:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205106.050205:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205106.050211:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205106.050223:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443139456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205106.050248:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205106/real 1788205106] req@ffff9023450ba300 x1875064443139072/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205106.050268:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443139072/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205106.050277:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205106.050281:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205106.050285:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443139072:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788205106.050306:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443139200:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788205106.050329:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205106.050341:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205106/real 1788205106] req@ffff9023450b8000 x1875064443139200/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205106.050352:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443139200/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205106.050359:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205106.050362:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205106.050365:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443139200:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788205106.050371:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443139328:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205106.050389:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205106.050400:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205106/real 1788205106] req@ffff9023450b9180 x1875064443139328/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205106.050412:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443139328/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205106.050419:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205106.050423:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205106.050426:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443139328:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788205106.050431:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443139456:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205106.050450:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205106.050462:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205106/real 1788205106] req@ffff9023450baa00 x1875064443139456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205106.050473:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443139456/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205106.050480:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205106.050483:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205106.050486:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443139456:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205106.881211:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205107.905541:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205108.929453:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205109.954283:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205110.980747:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205111.169159:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205111.169172:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205111.169177:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205111.169189:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4382 00000100:00080000:0.0:1788205111.169203:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205111.169211:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788205111.169258:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443139584/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205111.169352:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443139584:192.168.201.130@tcp:250:kworker.0 00000800:00000100:1.0:1788205111.169428:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788205111.169466:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205111.169471:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205111.169473:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205111.169478:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4382 00000100:00080000:0.0:1788205111.169482:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205111.169486:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205111.169497:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205111.169511:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443139712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205111.169764:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205111/real 1788205111] req@ffff9023450b9f80 x1875064443139584/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205111.169784:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443139584/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205111.169795:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205111.169799:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205111.169803:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443139584:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788205111.169829:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443139712:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788205111.169855:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205111.169866:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205111/real 1788205111] req@ffff9023450b9c00 x1875064443139712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205111.169898:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443139712/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205111.169906:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205111.169909:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205111.169913:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443139712:192.168.201.130@tcp:38:kworker.0 00000100:00080000:0.0:1788205111.170023:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205111.170027:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205111.170029:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205111.170034:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4382 00000100:00080000:0.0:1788205111.170038:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205111.170042:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205111.170048:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205111.170061:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443139840/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205111.170071:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205111.170074:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205111.170076:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205111.170079:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4382 00000100:00080000:0.0:1788205111.170082:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205111.170085:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205111.170089:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205111.170097:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443139968/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205111.170205:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443139840:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205111.170227:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205111.170238:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205111/real 1788205111] req@ffff9023450bad80 x1875064443139840/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205111.170249:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443139840/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205111.170258:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205111.170261:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205111.170264:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443139840:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788205111.170270:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443139968:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205111.170285:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205111.170293:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205111/real 1788205111] req@ffff9023450b8700 x1875064443139968/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205111.170310:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443139968/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205111.170318:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205111.170320:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205111.170323:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443139968:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205112.001090:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205113.025106:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205114.049300:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205115.074354:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205116.097055:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205116.289245:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205116.289252:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205116.289255:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205116.289263:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4387 00000100:00080000:1.0:1788205116.289272:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205116.289277:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788205116.289307:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519e300 x1875064443140096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205116.289336:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205116.289339:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205116.289341:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205116.289345:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4387 00000100:00080000:1.0:1788205116.289349:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205116.289353:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205116.289362:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205116.289371:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443140096:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788205116.289379:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443140224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205116.289389:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205116.289391:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788205116.289392:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205116.289395:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4387 00000100:00080000:1.0:1788205116.289397:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205116.289400:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205116.289404:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205116.289413:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443140352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205116.289419:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205116.289421:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788205116.289422:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205116.289424:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4387 00000100:00080000:1.0:1788205116.289427:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205116.289428:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205116.289431:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205116.289437:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443140480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205116.289447:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205116.289499:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205116/real 1788205116] req@ffff90234519e300 x1875064443140096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205116.289522:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519e300 x1875064443140096/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205116.289531:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205116.289535:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205116.289538:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443140096:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205116.289559:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443140224:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205116.289581:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205116.289594:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205116/real 1788205116] req@ffff90234519df80 x1875064443140224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205116.289615:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443140224/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205116.289621:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205116.289623:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205116.289625:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443140224:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205116.289629:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443140352:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205116.289644:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205116.289650:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205116/real 1788205116] req@ffff90234519d880 x1875064443140352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205116.289659:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443140352/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205116.289672:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205116.289675:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205116.289678:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443140352:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205116.289683:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443140480:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205116.289708:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205116.289720:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205116/real 1788205116] req@ffff90234519ed80 x1875064443140480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205116.289730:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443140480/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205116.289737:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205116.289740:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205116.289742:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443140480:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205117.121769:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205118.145062:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205119.169121:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205120.193155:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205121.217569:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205121.408981:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205121.408991:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205121.408995:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205121.409004:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4392 00000100:00080000:1.0:1788205121.409014:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205121.409021:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788205121.409049:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064443140608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205121.409069:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205121.409072:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205121.409073:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205121.409076:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4392 00000100:00080000:1.0:1788205121.409078:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205121.409080:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205121.409087:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205121.409094:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443140736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205121.409101:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205121.409103:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788205121.409104:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205121.409107:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4392 00000100:00080000:1.0:1788205121.409108:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205121.409110:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205121.409113:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205121.409118:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443140864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205121.409124:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443140608:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788205121.409130:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205121.409134:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788205121.409135:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205121.409139:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4392 00000100:00080000:1.0:1788205121.409143:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205121.409146:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205121.409151:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205121.409164:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443140992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205121.409199:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205121.409245:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205121/real 1788205121] req@ffff90234519f800 x1875064443140608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205121.409264:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064443140608/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205121.409282:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205121.409286:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205121.409290:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443140608:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205121.409297:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443140736:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205121.409319:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205121.409332:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205121/real 1788205121] req@ffff90234519fb80 x1875064443140736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205121.409343:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443140736/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205121.409351:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205121.409354:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205121.409357:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443140736:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205121.409377:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443140864:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205121.409406:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205121.409417:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205121/real 1788205121] req@ffff90234519c700 x1875064443140864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205121.409428:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443140864/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205121.409435:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205121.409438:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205121.409441:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443140864:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205121.409447:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443140992:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205121.409465:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205121.409475:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205121/real 1788205121] req@ffff90234519c000 x1875064443140992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205121.409496:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443140992/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205121.409504:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205121.409506:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205121.409509:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443140992:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788205122.241198:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205123.265006:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205124.289023:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205125.313243:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205126.337077:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205126.528986:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205126.528997:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205126.529002:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205126.529012:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4398 00000100:00080000:1.0:1788205126.529022:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205126.529030:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788205126.529067:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064443141120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205126.529095:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205126.529098:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205126.529100:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205126.529104:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4398 00000100:00080000:1.0:1788205126.529108:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205126.529113:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205126.529121:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205126.529134:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443141248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205126.529144:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205126.529149:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788205126.529150:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443141120:192.168.201.130@tcp:250:kworker.0 00000100:00080000:1.0:1788205126.529152:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205126.529162:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4398 00000100:00080000:1.0:1788205126.529165:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205126.529169:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205126.529173:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205126.529184:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443141376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205126.529194:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205126.529197:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788205126.529199:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205126.529203:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4398 00000100:00080000:1.0:1788205126.529208:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205126.529212:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205126.529216:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788205126.529222:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205126.529228:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443141504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205126.529263:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205126/real 1788205126] req@ffff90234519dc00 x1875064443141120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205126.529279:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064443141120/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205126.529286:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205126.529288:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205126.529291:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443141120:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205126.529308:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443141248:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205126.529322:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205126.529329:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205126/real 1788205126] req@ffff90234519f100 x1875064443141248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205126.529336:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443141248/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205126.529344:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205126.529346:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205126.529348:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443141248:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205126.529351:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443141376:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205126.529361:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205126.529366:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205126/real 1788205126] req@ffff90234519c380 x1875064443141376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205126.529380:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443141376/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205126.529387:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205126.529390:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205126.529393:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443141376:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205126.529399:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443141504:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205126.529417:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205126.529452:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205126/real 1788205126] req@ffff90234519ea00 x1875064443141504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205126.529464:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443141504/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205126.529471:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205126.529473:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205126.529475:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443141504:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205127.361056:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205128.385030:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205129.409208:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205130.433051:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205131.457048:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205131.649000:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205131.649008:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205131.649012:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205131.649021:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4403 00000100:00080000:2.0:1788205131.649030:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205131.649038:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205131.649074:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443141632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205131.649103:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205131.649107:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205131.649109:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205131.649114:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4403 00000100:00080000:2.0:1788205131.649119:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205131.649124:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205131.649133:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205131.649147:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443141760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205131.649153:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443141632:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788205131.649164:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205131.649169:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205131.649171:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205131.649178:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4403 00000100:00080000:2.0:1788205131.649182:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205131.649186:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205131.649192:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205131.649204:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443141888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788205131.649206:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788205131.649237:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205131.649241:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205131.649243:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205131.649247:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4403 00000100:00080000:2.0:1788205131.649250:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205131.649254:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205131.649258:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205131.649270:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443142016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205131.649383:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205131/real 1788205131] req@ffff90234598f480 x1875064443141632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205131.649397:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443141632/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205131.649431:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205131.649435:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205131.649439:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443141632:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788205131.649466:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443141760:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788205131.649482:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205131.649490:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205131/real 1788205131] req@ffff90234598ce00 x1875064443141760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205131.649499:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443141760/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205131.649513:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205131.649516:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205131.649520:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443141760:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788205131.649526:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443141888:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788205131.649545:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205131.649555:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205131/real 1788205131] req@ffff90234598f800 x1875064443141888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205131.649568:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443141888/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205131.649573:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205131.649575:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205131.649577:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443141888:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788205131.649581:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443142016:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788205131.649592:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205131.649598:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205131/real 1788205131] req@ffff90234598dc00 x1875064443142016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205131.649604:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443142016/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205131.649608:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205131.649610:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205131.649611:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443142016:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205132.481046:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205133.505034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205134.529055:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205135.555359:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205136.578967:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205136.770936:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205136.770948:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205136.770952:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205136.770961:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4408 00000100:00080000:2.0:1788205136.770972:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205136.770979:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205136.771018:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064443142144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205136.771049:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205136.771054:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205136.771056:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205136.771060:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4408 00000100:00080000:2.0:1788205136.771064:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205136.771068:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205136.771077:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205136.771090:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064443142272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205136.771101:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205136.771104:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205136.771106:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205136.771111:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4408 00000100:00100000:3.0:1788205136.771115:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443142144:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788205136.771116:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205136.771123:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205136.771129:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205136.771143:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064443142400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205136.771153:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205136.771157:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205136.771159:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205136.771163:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4408 00000100:00080000:2.0:1788205136.771175:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205136.771179:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000800:00000100:3.0:1788205136.771185:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000008:00100000:2.0:1788205136.771186:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205136.771205:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443142528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205136.771243:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205136/real 1788205136] req@ffff90234598d880 x1875064443142144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205136.771288:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064443142144/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205136.771299:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205136.771303:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205136.771306:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443142144:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788205136.771313:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443142272:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788205136.771349:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205136.771361:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205136/real 1788205136] req@ffff90234598e680 x1875064443142272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205136.771379:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064443142272/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205136.771387:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205136.771390:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205136.771393:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443142272:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788205136.771417:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443142400:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788205136.771451:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205136.771462:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205136/real 1788205136] req@ffff90234598ca80 x1875064443142400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205136.771485:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064443142400/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205136.771493:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205136.771496:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205136.771500:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443142400:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788205136.771505:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443142528:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788205136.771524:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205136.771535:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205136/real 1788205136] req@ffff90234598dc00 x1875064443142528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205136.771558:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443142528/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205136.771571:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205136.771584:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205136.771587:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443142528:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205137.601090:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205138.625054:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205139.649065:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205140.673250:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205141.697217:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205141.889296:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205141.889307:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205141.889311:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205141.889321:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4413 00000100:00080000:3.0:1788205141.889332:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205141.889339:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788205141.889374:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443142656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205141.889405:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205141.889409:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205141.889411:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205141.889416:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4413 00000100:00080000:3.0:1788205141.889420:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205141.889424:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205141.889435:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205141.889449:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443142784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205141.889454:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443142656:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788205141.889466:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205141.889469:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205141.889472:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205141.889476:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4413 00000100:00080000:3.0:1788205141.889479:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205141.889483:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205141.889487:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205141.889499:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443142912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205141.889509:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205141.889512:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205141.889514:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205141.889518:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4413 00000100:00080000:3.0:1788205141.889521:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788205141.889528:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788205141.889536:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205141.889542:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205141.889556:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443143040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205141.889575:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205141/real 1788205141] req@ffff902352689c00 x1875064443142656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205141.889593:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443142656/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205141.889603:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205141.889606:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205141.889610:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443142656:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205141.889630:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443142784:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205141.889648:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205141.889659:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205141/real 1788205141] req@ffff902352688000 x1875064443142784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205141.889669:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443142784/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205141.889676:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205141.889678:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205141.889681:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443142784:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205141.889686:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443142912:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205141.889701:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205141.889710:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205141/real 1788205141] req@ffff90235268b800 x1875064443142912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205141.889720:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443142912/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205141.889727:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205141.889730:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205141.889733:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443142912:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205141.889738:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443143040:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205141.889755:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205141.889766:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205141/real 1788205141] req@ffff90235268a300 x1875064443143040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205141.889777:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443143040/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205141.889784:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205141.889787:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205141.889790:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443143040:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788205142.721065:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205143.745104:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205144.769129:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205145.793344:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205146.818094:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205147.009040:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205147.009058:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205147.009063:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205147.009083:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4418 00000100:00080000:2.0:1788205147.009093:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205147.009100:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205147.009135:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443143168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205147.009161:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205147.009165:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205147.009166:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205147.009171:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4418 00000100:00080000:2.0:1788205147.009183:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205147.009188:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205147.009197:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205147.009208:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443143168:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788205147.009212:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443143296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205147.009230:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205147.009234:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205147.009236:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205147.009240:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4418 00000100:00080000:2.0:1788205147.009243:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205147.009248:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205147.009252:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205147.009263:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443143424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205147.009272:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788205147.009284:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205147.009287:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205147.009289:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205147.009294:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4418 00000100:00080000:2.0:1788205147.009297:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205147.009300:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788205147.009313:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205147/real 1788205147] req@ffff90234598f800 x1875064443143168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788205147.009317:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205147.009332:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443143552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205147.009337:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443143168/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205147.009367:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205147.009371:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205147.009375:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443143168:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205147.009397:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443143296:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205147.009417:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205147.009574:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205147/real 1788205147] req@ffff90234598ce00 x1875064443143296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205147.009586:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443143296/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205147.009593:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205147.009596:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205147.009599:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443143296:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205147.009604:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443143424:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205147.009623:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205147.009633:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205147/real 1788205147] req@ffff90234598f480 x1875064443143424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205147.009643:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443143424/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205147.009650:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205147.009653:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205147.009656:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443143424:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205147.009661:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443143552:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205147.009677:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205147.009686:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205147/real 1788205147] req@ffff90234598e300 x1875064443143552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205147.009695:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443143552/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205147.009701:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205147.009704:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205147.009706:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443143552:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205147.841049:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205148.865063:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205149.889055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205150.913071:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205151.937102:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205152.129027:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205152.129037:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205152.129040:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205152.129051:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4423 00000100:00080000:2.0:1788205152.129063:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205152.129071:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205152.129120:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443143680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205152.129163:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205152.129168:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205152.129170:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205152.129175:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4423 00000100:00080000:2.0:1788205152.129179:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205152.129185:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205152.129196:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205152.129211:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443143808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205152.129217:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443143680:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788205152.129226:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205152.129231:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205152.129233:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205152.129238:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4423 00000100:00080000:2.0:1788205152.129241:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205152.129245:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205152.129250:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205152.129262:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443143936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205152.129271:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205152.129275:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205152.129277:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205152.129280:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4423 00000100:00080000:2.0:1788205152.129284:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205152.129288:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205152.129292:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205152.129301:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064443144064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205152.129322:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205152.129390:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205152/real 1788205152] req@ffff90234598c380 x1875064443143680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205152.129411:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443143680/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205152.129424:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205152.129428:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205152.129432:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443143680:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205152.129463:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443143808:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205152.129495:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205152.129510:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205152/real 1788205152] req@ffff90234598d180 x1875064443143808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205152.129522:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443143808/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205152.129530:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205152.129533:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205152.129536:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443143808:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205152.129542:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443143936:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205152.129563:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205152.129576:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205152/real 1788205152] req@ffff90234598c700 x1875064443143936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205152.129587:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443143936/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205152.129596:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205152.129598:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205152.129602:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443143936:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205152.129608:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443144064:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205152.129626:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205152.129637:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205152/real 1788205152] req@ffff90234598f100 x1875064443144064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205152.129648:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064443144064/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205152.129662:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205152.129665:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205152.129669:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443144064:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205152.961101:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205153.985068:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205155.009066:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205156.033041:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205157.057057:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205157.249021:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205157.249033:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205157.249037:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205157.249047:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4428 00000100:00080000:1.0:1788205157.249058:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205157.249065:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788205157.249097:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f480 x1875064443144192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205157.249118:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205157.249120:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205157.249122:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205157.249125:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4428 00000100:00080000:1.0:1788205157.249129:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205157.249133:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205157.249142:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205157.249154:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443144192:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788205157.249159:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ea00 x1875064443144320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205157.249171:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205157.249175:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788205157.249177:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205157.249180:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4428 00000100:00080000:1.0:1788205157.249184:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205157.249187:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205157.249192:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205157.249205:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c380 x1875064443144448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205157.249215:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205157.249218:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788205157.249221:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788205157.249223:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788205157.249227:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4428 00000100:00080000:1.0:1788205157.249233:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205157.249238:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205157.249243:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205157.249256:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f100 x1875064443144576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205157.249280:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205157/real 1788205157] req@ffff90234519f480 x1875064443144192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205157.249298:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f480 x1875064443144192/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205157.249306:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205157.249308:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205157.249311:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443144192:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205157.249328:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443144320:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205157.249345:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205157.249353:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205157/real 1788205157] req@ffff90234519ea00 x1875064443144320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205157.249360:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ea00 x1875064443144320/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205157.249365:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205157.249366:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205157.249368:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443144320:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205157.249373:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443144448:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205157.249383:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205157.249388:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205157/real 1788205157] req@ffff90234519c380 x1875064443144448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205157.249395:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c380 x1875064443144448/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205157.249399:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205157.249401:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205157.249402:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443144448:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205157.249405:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443144576:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205157.249414:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205157.249433:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205157/real 1788205157] req@ffff90234519f100 x1875064443144576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205157.249445:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f100 x1875064443144576/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205157.249452:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205157.249455:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205157.249458:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443144576:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205158.081035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205159.105093:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205160.129017:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205161.154913:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205162.177093:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205162.370998:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205162.371007:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205162.371012:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205162.371022:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4433 00000100:00080000:3.0:1788205162.371033:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205162.371040:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788205162.371078:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064443144704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205162.371117:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205162.371121:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205162.371122:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205162.371126:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4433 00000100:00080000:3.0:1788205162.371130:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205162.371133:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205162.371142:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205162.371157:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443144832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205162.371168:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788205162.371171:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443144704:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788205162.371172:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205162.371176:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205162.371183:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4433 00000100:00080000:3.0:1788205162.371187:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205162.371191:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205162.371195:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205162.371207:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443144960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205162.371216:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205162.371220:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205162.371222:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205162.371226:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4433 00000100:00080000:3.0:1788205162.371230:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205162.371233:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205162.371237:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788205162.371246:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205162.371250:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443145088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205162.371303:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205162/real 1788205162] req@ffff90235268b100 x1875064443144704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205162.371316:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064443144704/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205162.371324:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205162.371326:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205162.371329:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443144704:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205162.371347:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443144832:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205162.371369:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205162.371378:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205162/real 1788205162] req@ffff90235268a300 x1875064443144832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205162.371385:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443144832/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205162.371390:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205162.371392:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205162.371394:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443144832:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205162.371398:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443144960:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205162.371416:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205162.371427:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205162/real 1788205162] req@ffff90235268b800 x1875064443144960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205162.371437:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443144960/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205162.371449:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205162.371451:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205162.371454:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443144960:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205162.371457:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443145088:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205162.371470:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205162.371476:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205162/real 1788205162] req@ffff902352688000 x1875064443145088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205162.371483:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443145088/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205162.371487:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205162.371489:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205162.371490:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443145088:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788205163.201171:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205164.225090:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205165.249076:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205166.273132:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205167.297043:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205167.488995:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205167.489006:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205167.489009:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205167.489018:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4439 00000100:00080000:3.0:1788205167.489029:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205167.489035:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788205167.489072:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443145216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205167.489110:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205167.489114:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205167.489116:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205167.489237:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4439 00000100:00080000:3.0:1788205167.489242:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205167.489246:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205167.489254:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205167.489267:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443145344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205167.489277:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205167.489282:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205167.489284:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205167.489288:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4439 00000100:00080000:3.0:1788205167.489291:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205167.489295:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205167.489300:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205167.489310:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443145472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205167.489320:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205167.489322:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205167.489324:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205167.489327:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4439 00000100:00080000:3.0:1788205167.489330:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205167.489333:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205167.489337:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205167.489344:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443145600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205167.489407:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443145216:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205167.489516:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443145344:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205167.489542:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443145472:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205167.489576:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443145600:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205168.321224:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205169.345503:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205170.369082:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788205170.561282:0:2349:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788205170.561341:0:2349:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788205170.561363:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788205170.561411:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788205170.561436:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788205170.561451:0:2349:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:0.0:1788205170.561459:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205167/real 1788205170] req@ffff902352689c00 x1875064443145216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205170.561500:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443145216/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205170.561514:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205170.561519:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205170.561524:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443145216:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205170.561532:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205167/real 1788205170] req@ffff902352688e00 x1875064443145344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205170.561544:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443145344/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205170.561552:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205170.561555:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205170.561559:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443145344:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205170.561566:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205167/real 1788205170] req@ffff90235268a680 x1875064443145472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205170.561592:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443145472/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205170.561600:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205170.561603:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205170.561606:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443145472:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205170.561613:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205167/real 1788205170] req@ffff902352689500 x1875064443145600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205170.561624:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443145600/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205170.561631:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205170.561634:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205170.561637:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443145600:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788205171.393054:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205172.417041:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205172.608953:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205172.608963:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205172.608966:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205172.608976:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4444 00000100:00080000:3.0:1788205172.608987:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205172.608994:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788205172.609031:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443145728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205172.609060:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205172.609064:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205172.609066:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205172.609071:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4444 00000100:00080000:3.0:1788205172.609075:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205172.609080:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205172.609089:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205172.609101:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443145728:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788205172.609104:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443145856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205172.609117:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205172.609120:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205172.609122:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205172.609125:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4444 00000100:00080000:3.0:1788205172.609129:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205172.609133:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205172.609138:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205172.609149:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443145984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205172.609167:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788205172.609168:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205172.609173:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205172.609175:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205172.609179:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4444 00000100:00080000:3.0:1788205172.609184:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205172.609188:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205172.609192:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205172.609202:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443146112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205172.609212:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205172/real 1788205172] req@ffff902352689180 x1875064443145728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205172.609232:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443145728/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205172.609243:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205172.609245:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205172.609248:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443145728:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205172.609265:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443145856:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205172.609283:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205172.609293:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205172/real 1788205172] req@ffff902352688700 x1875064443145856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205172.609300:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443145856/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205172.609305:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205172.609307:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205172.609309:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443145856:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205172.609313:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443145984:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205172.609323:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205172.609329:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205172/real 1788205172] req@ffff90235268bb80 x1875064443145984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205172.609336:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443145984/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205172.609341:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205172.609343:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205172.609345:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443145984:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205172.609348:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443146112:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205172.609357:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205172.609362:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205172/real 1788205172] req@ffff902352689880 x1875064443146112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205172.609368:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443146112/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205172.609372:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205172.609374:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205172.609375:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443146112:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205173.441058:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205174.465057:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205175.489124:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205176.513109:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205177.537075:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205177.728978:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205177.728985:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205177.728988:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205177.728996:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4449 00000100:00080000:3.0:1788205177.729004:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205177.729009:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788205177.729040:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443146240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205177.729063:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205177.729066:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205177.729068:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205177.729072:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4449 00000100:00080000:3.0:1788205177.729075:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205177.729078:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205177.729087:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205177.729099:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064443146368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205177.729110:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788205177.729113:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443146240:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788205177.729119:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205177.729123:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205177.729126:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4449 00000100:00080000:3.0:1788205177.729129:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205177.729131:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205177.729136:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205177.729147:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443146496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205177.729156:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205177.729159:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205177.729161:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205177.729165:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4449 00000100:00080000:3.0:1788205177.729169:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205177.729172:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205177.729176:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788205177.729177:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205177.729188:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443146624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205177.729232:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205177/real 1788205177] req@ffff90235268aa00 x1875064443146240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205177.729247:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443146240/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205177.729255:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205177.729257:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205177.729260:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443146240:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205177.729277:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443146368:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205177.729292:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205177.729300:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205177/real 1788205177] req@ffff902352688a80 x1875064443146368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205177.729307:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064443146368/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205177.729312:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205177.729314:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205177.729315:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443146368:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205177.729319:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443146496:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205177.729328:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205177.729334:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205177/real 1788205177] req@ffff90235268ad80 x1875064443146496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205177.729344:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443146496/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205177.729370:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205177.729373:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205177.729376:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443146496:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205177.729382:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443146624:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205177.729399:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205177.729409:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205177/real 1788205177] req@ffff90235268b480 x1875064443146624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205177.729419:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443146624/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205177.729426:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205177.729429:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205177.729432:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443146624:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205178.561060:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205179.588599:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205180.609116:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205181.633429:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205182.657080:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205182.849022:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205182.849032:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205182.849035:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205182.849056:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4454 00000100:00080000:2.0:1788205182.849067:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205182.849076:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205182.849113:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ed80 x1875064443146752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205182.849141:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205182.849144:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205182.849146:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205182.849150:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4454 00000100:00080000:2.0:1788205182.849154:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205182.849159:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205182.849169:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205182.849172:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443146752:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788205182.849185:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598fb80 x1875064443146880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205182.849196:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205182.849200:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205182.849202:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205182.849206:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4454 00000100:00080000:2.0:1788205182.849210:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205182.849214:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205182.849219:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788205182.849231:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788205182.849235:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598df80 x1875064443147008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205182.849246:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205182.849250:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205182.849251:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205182.849255:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4454 00000100:00080000:2.0:1788205182.849259:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205182.849263:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205182.849269:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205182.849271:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205182/real 1788205182] req@ffff90234598ed80 x1875064443146752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205182.849283:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064443147136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205182.849291:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ed80 x1875064443146752/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205182.849304:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205182.849308:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205182.849313:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443146752:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205182.849334:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443146880:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205182.849352:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205182.849362:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205182/real 1788205182] req@ffff90234598fb80 x1875064443146880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205182.849372:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598fb80 x1875064443146880/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205182.849380:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205182.849382:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205182.849386:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443146880:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205182.849391:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443147008:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205182.849405:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205182.849413:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205182/real 1788205182] req@ffff90234598df80 x1875064443147008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205182.849423:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598df80 x1875064443147008/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205182.849441:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205182.849444:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205182.849447:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443147008:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205182.849476:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443147136:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205182.849493:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205182.849502:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205182/real 1788205182] req@ffff90234598f100 x1875064443147136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205182.849522:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064443147136/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205182.849529:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205182.849532:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205182.849535:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443147136:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788205183.681147:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205184.705083:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205185.729059:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205186.753080:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205187.777104:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205187.969091:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205187.969101:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205187.969105:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205187.969115:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4459 00000100:00080000:0.0:1788205187.969126:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205187.969133:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788205187.969174:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443147264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205187.969213:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205187.969217:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205187.969219:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205187.969223:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4459 00000100:00080000:0.0:1788205187.969227:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205187.969230:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205187.969239:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205187.969251:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443147392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205187.969266:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205187.969269:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205187.969270:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205187.969274:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4459 00000100:00080000:0.0:1788205187.969277:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205187.969281:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205187.969285:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205187.969295:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443147520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205187.969303:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205187.969306:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205187.969307:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205187.969311:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4459 00000100:00080000:0.0:1788205187.969314:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205187.969317:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205187.969321:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205187.969330:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443147648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205187.969431:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443147264:192.168.201.130@tcp:250:kworker.0 00000800:00000100:3.0:1788205187.969487:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205187.969532:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205187/real 1788205187] req@ffff9023450b8380 x1875064443147264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205187.969553:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443147264/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205187.969569:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205187.969572:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205187.969576:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443147264:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788205187.969582:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443147392:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788205187.969603:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205187.969615:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205187/real 1788205187] req@ffff9023450b8a80 x1875064443147392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205187.969626:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443147392/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205187.969633:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205187.969636:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205187.969639:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443147392:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788205187.969645:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443147520:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788205187.969664:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205187.969674:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205187/real 1788205187] req@ffff9023450bb800 x1875064443147520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205187.969684:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443147520/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205187.969691:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205187.969694:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205187.969698:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443147520:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788205187.969703:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443147648:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788205187.969718:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205187.969727:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205187/real 1788205187] req@ffff9023450b9500 x1875064443147648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205187.969739:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443147648/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205187.969745:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205187.969748:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205187.969751:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443147648:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205188.801082:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205189.825043:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205190.849022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205191.873020:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205192.897031:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205193.089060:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205193.089069:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205193.089073:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205193.089082:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4464 00000100:00080000:1.0:1788205193.089094:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205193.089101:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788205193.089201:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519dc00 x1875064443147776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205193.089234:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205193.089238:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205193.089240:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205193.089244:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4464 00000100:00080000:1.0:1788205193.089248:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205193.089252:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205193.089262:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205193.089307:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c000 x1875064443147904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205193.089317:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205193.089320:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788205193.089322:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205193.089325:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4464 00000100:00080000:1.0:1788205193.089329:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205193.089332:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205193.089337:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205193.089348:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519c700 x1875064443148032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205193.089357:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205193.089373:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788205193.089374:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205193.089378:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4464 00000100:00080000:1.0:1788205193.089380:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205193.089384:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205193.089388:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205193.089400:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519fb80 x1875064443148160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205193.089492:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443147776:192.168.201.130@tcp:250:kworker.0 00000800:00000100:3.0:1788205193.089547:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205193.089590:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205193/real 1788205193] req@ffff90234519dc00 x1875064443147776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205193.089607:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519dc00 x1875064443147776/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205193.089617:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205193.089621:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205193.089625:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443147776:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788205193.089632:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443147904:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788205193.089652:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205193.089664:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205193/real 1788205193] req@ffff90234519c000 x1875064443147904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205193.089676:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c000 x1875064443147904/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205193.089684:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205193.089687:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205193.089690:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443147904:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788205193.089695:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443148032:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788205193.089714:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205193.089725:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205193/real 1788205193] req@ffff90234519c700 x1875064443148032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205193.089736:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519c700 x1875064443148032/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205193.089743:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205193.089746:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205193.089749:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443148032:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788205193.089754:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443148160:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788205193.089772:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205193.089782:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205193/real 1788205193] req@ffff90234519fb80 x1875064443148160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205193.089793:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519fb80 x1875064443148160/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205193.089805:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205193.089808:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205193.089811:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443148160:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205193.921131:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205194.945082:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205195.969156:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205196.994497:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205198.017038:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205198.209071:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205198.209086:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205198.209090:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205198.209098:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4469 00000100:00080000:3.0:1788205198.209107:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205198.209113:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788205198.209145:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688380 x1875064443148288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205198.209233:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205198.209236:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205198.209238:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205198.209242:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4469 00000100:00080000:3.0:1788205198.209245:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205198.209247:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205198.209255:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205198.209268:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b480 x1875064443148416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205198.209275:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205198.209277:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205198.209278:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205198.209281:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4469 00000100:00080000:3.0:1788205198.209283:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205198.209285:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205198.209288:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205198.209295:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268ad80 x1875064443148544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205198.209303:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205198.209305:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205198.209306:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205198.209309:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4469 00000100:00080000:3.0:1788205198.209312:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205198.209316:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205198.209320:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205198.209330:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688a80 x1875064443148672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205198.209408:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443148288:192.168.201.130@tcp:250:kworker.0 00000800:00000100:0.0:1788205198.209462:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205198.209509:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205198/real 1788205198] req@ffff902352688380 x1875064443148288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205198.209532:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688380 x1875064443148288/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205198.209542:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205198.209546:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205198.209550:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443148288:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205198.209557:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443148416:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205198.209578:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205198.209590:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205198/real 1788205198] req@ffff90235268b480 x1875064443148416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205198.209602:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b480 x1875064443148416/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205198.209610:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205198.209613:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205198.209616:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443148416:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205198.209621:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443148544:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205198.209640:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205198.209651:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205198/real 1788205198] req@ffff90235268ad80 x1875064443148544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205198.209662:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268ad80 x1875064443148544/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205198.209676:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205198.209679:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205198.209682:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443148544:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205198.209687:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443148672:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205198.209701:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205198.209708:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205198/real 1788205198] req@ffff902352688a80 x1875064443148672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205198.209714:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688a80 x1875064443148672/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205198.209719:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205198.209721:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205198.209723:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443148672:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205199.041113:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205200.065558:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205201.089098:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205202.113075:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205203.137043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205203.328993:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205203.329002:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205203.329006:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205203.329016:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4474 00000100:00080000:3.0:1788205203.329025:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205203.329032:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788205203.329070:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268aa00 x1875064443148800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205203.329136:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205203.329141:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205203.329143:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205203.329148:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4474 00000100:00080000:3.0:1788205203.329153:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788205203.329158:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443148800:192.168.201.130@tcp:250:kworker.0 00000100:00080000:3.0:1788205203.329160:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205203.329174:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205203.329190:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689880 x1875064443148928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205203.329201:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205203.329205:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205203.329207:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205203.329211:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4474 00000100:00080000:3.0:1788205203.329215:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788205203.329219:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:3.0:1788205203.329224:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205203.329230:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205203.329243:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268bb80 x1875064443149056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205203.329253:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205203.329256:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788205203.329260:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205203/real 1788205203] req@ffff90235268aa00 x1875064443148800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205203.329260:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205203.329269:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4474 00000100:00080000:3.0:1788205203.329273:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205203.329278:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205203.329285:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205203.329292:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268aa00 x1875064443148800/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205203.329298:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688700 x1875064443149184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205203.329310:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205203.329315:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205203.329319:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443148800:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205203.329340:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443148928:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205203.329359:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205203.329367:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205203/real 1788205203] req@ffff902352689880 x1875064443148928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205203.329375:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689880 x1875064443148928/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205203.329380:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205203.329382:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205203.329384:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443148928:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205203.329388:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443149056:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205203.329404:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205203.329415:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205203/real 1788205203] req@ffff90235268bb80 x1875064443149056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205203.329427:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268bb80 x1875064443149056/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205203.329435:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205203.329439:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205203.329442:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443149056:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205203.329447:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443149184:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205203.329465:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205203.329481:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205203/real 1788205203] req@ffff902352688700 x1875064443149184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205203.329489:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688700 x1875064443149184/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205203.329493:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205203.329495:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205203.329497:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443149184:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788205204.161032:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205205.185048:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205206.209026:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205207.233073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205208.257034:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205208.449002:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205208.449026:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205208.449031:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205208.449040:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4479 00000100:00080000:2.0:1788205208.449050:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205208.449057:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205208.449096:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443149312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205208.449126:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205208.449130:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205208.449132:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205208.449136:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4479 00000100:00080000:2.0:1788205208.449140:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205208.449144:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205208.449154:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205208.449173:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443149312:192.168.201.130@tcp:250:kworker.0 00000100:00100000:2.0:1788205208.449176:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443149440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205208.449188:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205208.449190:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205208.449192:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205208.449195:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4479 00000100:00080000:2.0:1788205208.449198:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205208.449201:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205208.449205:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205208.449215:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443149568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205208.449220:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205208.449223:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205208.449224:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205208.449226:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4479 00000100:00080000:2.0:1788205208.449230:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205208.449232:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205208.449234:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205208.449242:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443149696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205208.449245:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205208.449294:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205208/real 1788205208] req@ffff90234598c700 x1875064443149312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205208.449311:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443149312/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205208.449321:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205208.449325:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205208.449336:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443149312:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205208.449362:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443149440:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205208.449383:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205208.449394:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205208/real 1788205208] req@ffff90234598d180 x1875064443149440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205208.449405:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443149440/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205208.449417:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205208.449420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205208.449423:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443149440:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205208.449429:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443149568:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205208.449445:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205208.449452:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205208/real 1788205208] req@ffff90234598c380 x1875064443149568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205208.449459:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443149568/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205208.449464:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205208.449466:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205208.449468:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443149568:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205208.449471:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443149696:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205208.449480:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205208.449485:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205208/real 1788205208] req@ffff90234598e300 x1875064443149696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205208.449491:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443149696/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205208.449495:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205208.449497:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205208.449499:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443149696:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205209.281061:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205210.305008:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205211.329056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205212.353333:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205213.377285:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205213.573672:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205213.573686:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205213.573691:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205213.573704:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4485 00000100:00080000:3.0:1788205213.573717:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205213.573725:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788205213.573772:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689180 x1875064443149824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205213.573814:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205213.573819:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205213.573820:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205213.573825:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4485 00000100:00080000:3.0:1788205213.573829:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205213.573833:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205213.573843:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205213.573855:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689500 x1875064443149952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205213.573864:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205213.573870:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205213.573871:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205213.573914:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4485 00000100:00080000:3.0:1788205213.573927:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205213.573932:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205213.573937:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205213.573949:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a680 x1875064443150080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205213.573958:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205213.573962:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205213.573963:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205213.573967:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4485 00000100:00080000:3.0:1788205213.573970:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205213.573974:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205213.573979:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205213.573989:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688e00 x1875064443150208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205213.574100:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443149824:192.168.201.130@tcp:250:kworker.0 00000800:00000100:0.0:1788205213.574189:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205213.574304:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205213/real 1788205213] req@ffff902352689180 x1875064443149824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205213.574326:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689180 x1875064443149824/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205213.574337:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205213.574341:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205213.574345:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443149824:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205213.574353:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443149952:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205213.574376:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205213.574388:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205213/real 1788205213] req@ffff902352689500 x1875064443149952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205213.574400:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689500 x1875064443149952/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205213.574409:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205213.574420:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205213.574424:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443149952:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205213.574433:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443150080:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205213.574458:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205213.574470:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205213/real 1788205213] req@ffff90235268a680 x1875064443150080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205213.574487:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a680 x1875064443150080/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205213.574494:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205213.574497:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205213.574501:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443150080:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205213.574506:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443150208:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205213.574523:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205213.574650:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205213/real 1788205213] req@ffff902352688e00 x1875064443150208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205213.574664:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688e00 x1875064443150208/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205213.574671:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205213.574674:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205213.574677:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443150208:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205214.401103:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205215.425000:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205216.449068:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205217.474132:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205218.497468:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205218.689941:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205218.689953:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205218.689958:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205218.689969:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4490 00000100:00080000:1.0:1788205218.689980:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205218.689987:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788205218.690024:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519f800 x1875064443150336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205218.690052:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205218.690056:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205218.690058:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205218.690063:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4490 00000100:00080000:1.0:1788205218.690067:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205218.690072:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205218.690082:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205218.690084:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443150336:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788205218.690097:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519ed80 x1875064443150464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205218.690109:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205218.690112:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788205218.690115:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205218.690119:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4490 00000100:00080000:1.0:1788205218.690122:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205218.690127:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205218.690131:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205218.690143:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519d880 x1875064443150592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205218.690151:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:1.0:1788205218.690157:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205218.690160:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788205218.690162:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205218.690167:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4490 00000100:00080000:1.0:1788205218.690172:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205218.690176:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:1.0:1788205218.690180:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205218.690192:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234519df80 x1875064443150720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205218.690211:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205218/real 1788205218] req@ffff90234519f800 x1875064443150336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205218.690227:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519f800 x1875064443150336/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205218.690236:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205218.690239:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205218.690242:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443150336:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205218.690260:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443150464:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205218.690279:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205218.690292:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205218/real 1788205218] req@ffff90234519ed80 x1875064443150464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205218.690305:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519ed80 x1875064443150464/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205218.690314:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205218.690318:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205218.690321:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443150464:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205218.690327:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443150592:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205218.690346:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205218.690356:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205218/real 1788205218] req@ffff90234519d880 x1875064443150592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205218.690368:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519d880 x1875064443150592/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205218.690375:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205218.690378:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205218.690381:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443150592:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205218.690387:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443150720:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205218.690403:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205218.690412:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205218/real 1788205218] req@ffff90234519df80 x1875064443150720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205218.690423:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234519df80 x1875064443150720/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205218.690443:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205218.690446:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205218.690449:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234519df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443150720:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205219.521085:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205220.545830:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205221.569075:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205222.593165:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205223.620343:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205223.808985:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205223.808998:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205223.809004:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205223.809016:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4495 00000100:00080000:3.0:1788205223.809030:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205223.809038:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788205223.809078:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352689c00 x1875064443150848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205223.809113:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205223.809117:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205223.809119:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205223.809124:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4495 00000100:00080000:3.0:1788205223.809128:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205223.809132:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205223.809141:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205223.809154:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443150976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205223.809164:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205223.809167:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205223.809169:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205223.809172:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4495 00000100:00080000:3.0:1788205223.809176:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205223.809180:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205223.809184:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205223.809196:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443151104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205223.809204:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205223.809207:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205223.809209:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205223.809213:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4495 00000100:00080000:3.0:1788205223.809217:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205223.809220:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205223.809224:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205223.809234:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443151232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205223.809324:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443150848:192.168.201.130@tcp:250:kworker.0 00000800:00000100:0.0:1788205223.809389:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205223.809441:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205223/real 1788205223] req@ffff902352689c00 x1875064443150848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205223.809458:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352689c00 x1875064443150848/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205223.809468:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205223.809472:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205223.809475:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352689c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443150848:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205223.809482:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443150976:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205223.809504:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205223.809516:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205223/real 1788205223] req@ffff902352688000 x1875064443150976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205223.809527:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443150976/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205223.809534:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205223.809537:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205223.809540:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443150976:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205223.809546:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443151104:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205223.809564:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205223.809574:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205223/real 1788205223] req@ffff90235268b800 x1875064443151104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205223.809585:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443151104/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205223.809592:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205223.809595:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205223.809598:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443151104:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205223.809603:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443151232:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205223.809621:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205223.809631:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205223/real 1788205223] req@ffff90235268a300 x1875064443151232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205223.809641:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443151232/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205223.809654:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205223.809657:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205223.809660:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443151232:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788205224.641220:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205225.665105:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205226.690239:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205227.713458:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205228.737176:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205228.929303:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205228.929314:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205228.929318:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205228.929328:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4500 00000100:00080000:3.0:1788205228.929338:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205228.929345:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:3.0:1788205228.929395:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b100 x1875064443151360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205228.929430:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205228.929434:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205228.929436:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205228.929441:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4500 00000100:00080000:3.0:1788205228.929445:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205228.929450:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205228.929461:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205228.929475:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268a300 x1875064443151488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205228.929484:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205228.929490:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205228.929492:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205228.929495:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4500 00000100:00080000:3.0:1788205228.929498:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205228.929501:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205228.929506:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205228.929515:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90235268b800 x1875064443151616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205228.929524:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205228.929527:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205228.929529:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205228.929532:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4500 00000100:00080000:3.0:1788205228.929535:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205228.929539:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:3.0:1788205228.929543:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205228.929551:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff902352688000 x1875064443151744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205228.929673:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443151360:192.168.201.130@tcp:250:kworker.0 00000800:00000100:0.0:1788205228.929729:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205228.929774:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205228/real 1788205228] req@ffff90235268b100 x1875064443151360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205228.929788:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b100 x1875064443151360/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205228.929796:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205228.929799:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205228.929802:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443151360:192.168.201.130@tcp:250:kworker.0 00000100:00100000:0.0:1788205228.929806:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443151488:192.168.201.130@tcp:38:kworker.0 00000800:00000100:0.0:1788205228.929821:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205228.929829:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205228/real 1788205228] req@ffff90235268a300 x1875064443151488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205228.929836:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268a300 x1875064443151488/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205228.929841:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205228.929843:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205228.929845:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443151488:192.168.201.130@tcp:38:kworker.0 00000100:00100000:0.0:1788205228.929849:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443151616:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205228.929859:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205228.929865:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205228/real 1788205228] req@ffff90235268b800 x1875064443151616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205228.929871:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90235268b800 x1875064443151616/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205228.929896:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205228.929897:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205228.929899:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90235268b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443151616:192.168.201.130@tcp:8:kworker.0 00000100:00100000:0.0:1788205228.929903:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443151744:192.168.201.130@tcp:8:kworker.0 00000800:00000100:0.0:1788205228.929915:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205228.929921:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205228/real 1788205228] req@ffff902352688000 x1875064443151744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205228.929927:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff902352688000 x1875064443151744/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205228.929932:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205228.929934:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:0.0:1788205228.929936:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff902352688000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443151744:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205229.761846:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205230.785631:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205231.809146:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205232.833667:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205233.857239:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205234.050097:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205234.050108:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205234.050111:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205234.050120:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4505 00000100:00080000:2.0:1788205234.050130:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205234.050137:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205234.050170:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443151872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205234.050201:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205234.050204:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205234.050205:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205234.050209:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4505 00000100:00080000:2.0:1788205234.050212:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205234.050215:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205234.050224:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205234.050238:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443152000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205234.050247:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205234.050249:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205234.050251:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205234.050254:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4505 00000100:00080000:2.0:1788205234.050257:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205234.050260:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205234.050264:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205234.050272:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443152128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205234.050279:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205234.050283:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205234.050284:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205234.050288:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4505 00000100:00080000:2.0:1788205234.050290:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205234.050293:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205234.050296:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205234.050303:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443152256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205234.050446:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443151872:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788205234.050539:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443152000:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788205234.051364:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443152128:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788205234.051407:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443152256:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205234.881098:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205235.905944:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205236.930744:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788205237.121308:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.130:988 00000400:00000100:2.0:1788205237.121363:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.130@tcp at host 192.168.201.130 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788205237.121385:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788205237.121440:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788205237.121452:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000800:00000100:2.0:1788205237.121461:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.130@tcp 00000100:00100000:3.0:1788205237.121664:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205234/real 1788205237] req@ffff90234598f480 x1875064443151872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205237.121687:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443151872/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205237.121698:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205237.121703:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205237.121708:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443151872:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788205237.121717:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205234/real 1788205237] req@ffff90234598ce00 x1875064443152000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205237.121731:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443152000/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205237.121740:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205237.121744:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205237.121746:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443152000:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788205237.121753:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205234/real 1788205237] req@ffff90234598f800 x1875064443152128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205237.121765:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443152128/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205237.121772:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205237.121776:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205237.121780:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443152128:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788205237.121787:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205234/real 1788205237] req@ffff90234598dc00 x1875064443152256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205237.121800:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443152256/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205237.121805:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205237.121808:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205237.121810:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443152256:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205237.953421:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205238.977177:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205239.169965:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205239.171532:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205239.171540:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205239.171552:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4510 00000100:00080000:0.0:1788205239.171565:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205239.171573:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788205239.171628:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443152384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205239.171668:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205239.171671:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205239.171673:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205239.171677:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4510 00000100:00080000:0.0:1788205239.171681:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205239.171685:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205239.171702:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205239.171715:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064443152512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205239.171725:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205239.171728:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205239.171730:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205239.171733:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4510 00000100:00080000:0.0:1788205239.171736:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205239.171740:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205239.171744:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205239.171754:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064443152640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205239.171768:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205239.171771:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205239.171772:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205239.171775:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4510 00000100:00080000:0.0:1788205239.171778:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205239.171782:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205239.171785:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205239.171793:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064443152768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205239.175031:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443152384:192.168.201.130@tcp:250:kworker.0 00000800:00000100:1.0:1788205239.175121:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205239.175163:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205239/real 1788205239] req@ffff9023450bb480 x1875064443152384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205239.175182:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443152384/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205239.175194:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205239.175198:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205239.175202:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443152384:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788205239.175221:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443152512:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788205239.175248:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205239.175260:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205239/real 1788205239] req@ffff9023450bbb80 x1875064443152512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205239.175271:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064443152512/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205239.175289:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205239.175293:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205239.175296:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443152512:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788205239.175305:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443152640:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205239.175327:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205239.177761:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205239/real 1788205239] req@ffff9023450b8e00 x1875064443152640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205239.177779:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064443152640/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205239.177787:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205239.177792:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205239.177796:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443152640:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788205239.177807:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443152768:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205239.177864:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205239.177891:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205239/real 1788205239] req@ffff9023450ba680 x1875064443152768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205239.177901:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064443152768/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205239.177905:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205239.177907:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205239.177909:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443152768:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205240.001161:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205241.025317:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205242.049192:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205243.076490:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205244.097135:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205244.289131:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205244.289141:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205244.289146:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205244.289156:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4515 00000100:00080000:0.0:1788205244.289166:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205244.289173:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788205244.289210:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba680 x1875064443152896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205244.289244:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205244.289248:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205244.289250:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205244.289254:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4515 00000100:00080000:0.0:1788205244.289258:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205244.289262:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205244.289273:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205244.289285:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8e00 x1875064443153024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205244.289295:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205244.289298:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205244.289300:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205244.289303:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4515 00000100:00080000:0.0:1788205244.289306:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205244.289310:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205244.289314:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205244.289323:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bbb80 x1875064443153152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205244.289332:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205244.289336:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205244.289337:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205244.289341:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4515 00000100:00080000:0.0:1788205244.289344:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205244.289347:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205244.289351:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205244.289359:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb480 x1875064443153280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205244.289471:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443152896:192.168.201.130@tcp:250:kworker.0 00000800:00000100:1.0:1788205244.289531:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205244.289581:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205244/real 1788205244] req@ffff9023450ba680 x1875064443152896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205244.289599:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba680 x1875064443152896/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205244.289610:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205244.289614:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205244.289618:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443152896:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788205244.289625:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443153024:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788205244.289648:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205244.289660:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205244/real 1788205244] req@ffff9023450b8e00 x1875064443153024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205244.289672:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8e00 x1875064443153024/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205244.289680:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205244.289683:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205244.289686:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443153024:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788205244.289692:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443153152:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205244.289709:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205244.289719:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205244/real 1788205244] req@ffff9023450bbb80 x1875064443153152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205244.289737:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bbb80 x1875064443153152/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205244.289745:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205244.289748:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205244.289751:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443153152:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788205244.290037:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443153280:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205244.290060:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205244.290072:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205244/real 1788205244] req@ffff9023450bb480 x1875064443153280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205244.290084:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb480 x1875064443153280/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205244.290092:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205244.290096:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205244.290099:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443153280:192.168.201.130@tcp:8:kworker.0 00010000:00010000:0.0:1788205245.121540:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205246.145219:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205247.169249:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205248.193271:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205249.217041:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205249.409007:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205249.409015:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205249.409018:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205249.409026:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4520 00000100:00080000:0.0:1788205249.409036:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205249.409047:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788205249.409084:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9500 x1875064443153408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205249.409123:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205249.409126:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205249.409128:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205249.409132:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4520 00000100:00080000:0.0:1788205249.409135:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205249.409139:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205249.409157:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205249.409169:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bb800 x1875064443153536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205249.409169:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443153408:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788205249.409184:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205249.409186:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205249.409187:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205249.409190:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4520 00000100:00080000:0.0:1788205249.409199:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205249.409202:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205249.409205:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205249.409213:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8a80 x1875064443153664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205249.409218:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205249.409221:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205249.409222:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205249.409225:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4520 00000100:00080000:0.0:1788205249.409233:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205249.409236:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205249.409238:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788205249.409246:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:0.0:1788205249.409252:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8380 x1875064443153792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205249.409316:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205249/real 1788205249] req@ffff9023450b9500 x1875064443153408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205249.409335:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9500 x1875064443153408/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205249.409345:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205249.409349:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205249.409352:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443153408:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788205249.409372:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443153536:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788205249.409391:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205249.409401:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205249/real 1788205249] req@ffff9023450bb800 x1875064443153536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205249.409412:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bb800 x1875064443153536/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205249.409419:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205249.409422:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205249.409425:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443153536:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788205249.409430:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443153664:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205249.409445:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205249.409453:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205249/real 1788205249] req@ffff9023450b8a80 x1875064443153664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205249.409463:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8a80 x1875064443153664/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205249.409470:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205249.409473:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205249.409476:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443153664:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788205249.409481:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443153792:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205249.409494:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205249.409502:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205249/real 1788205249] req@ffff9023450b8380 x1875064443153792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205249.409512:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8380 x1875064443153792/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205249.409518:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205249.409521:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205249.409524:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443153792:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205250.241096:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205251.265210:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205252.289160:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205253.313294:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205254.337013:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205254.528983:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205254.528990:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205254.528993:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205254.529000:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4526 00000100:00080000:2.0:1788205254.529008:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205254.529014:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205254.529046:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ca80 x1875064443153920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205254.529067:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205254.529069:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205254.529070:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205254.529073:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4526 00000100:00080000:2.0:1788205254.529075:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205254.529077:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205254.529083:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205254.529091:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e680 x1875064443154048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205254.529098:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205254.529102:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788205254.529103:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443153920:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788205254.529104:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205254.529112:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4526 00000100:00080000:2.0:1788205254.529115:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205254.529117:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205254.529121:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205254.529128:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d880 x1875064443154176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205254.529135:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205254.529137:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205254.529138:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205254.529140:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4526 00000100:00080000:2.0:1788205254.529143:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205254.529146:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205254.529149:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788205254.529158:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788205254.529159:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598dc00 x1875064443154304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205254.529191:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205254/real 1788205254] req@ffff90234598ca80 x1875064443153920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205254.529204:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ca80 x1875064443153920/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205254.529221:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205254.529226:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205254.529229:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443153920:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788205254.529248:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443154048:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788205254.529264:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205254.529272:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205254/real 1788205254] req@ffff90234598e680 x1875064443154048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205254.529280:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e680 x1875064443154048/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205254.529285:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205254.529287:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205254.529289:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443154048:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788205254.529292:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443154176:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205254.529301:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205254.529306:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205254/real 1788205254] req@ffff90234598d880 x1875064443154176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205254.529313:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d880 x1875064443154176/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205254.529317:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205254.529319:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205254.529320:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443154176:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788205254.529323:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443154304:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205254.529331:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205254.529336:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205254/real 1788205254] req@ffff90234598dc00 x1875064443154304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205254.529346:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598dc00 x1875064443154304/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205254.529351:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205254.529352:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205254.529354:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443154304:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788205255.361092:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205256.385014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205257.409041:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205258.433044:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205259.457907:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205259.648989:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205259.648999:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205259.649002:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205259.649011:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4531 00000100:00080000:2.0:1788205259.649022:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205259.649029:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205259.649066:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f800 x1875064443154432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205259.649098:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205259.649102:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205259.649104:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205259.649108:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4531 00000100:00080000:2.0:1788205259.649112:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205259.649115:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:1.0:1788205259.649165:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443154432:192.168.201.130@tcp:250:kworker.0 00000008:00100000:2.0:1788205259.649166:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205259.649184:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598ce00 x1875064443154560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205259.649196:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205259.649199:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205259.649201:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205259.649205:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4531 00000100:00080000:2.0:1788205259.649209:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205259.649212:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205259.649217:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205259.649227:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f480 x1875064443154688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205259.649234:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:2.0:1788205259.649239:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205259.649244:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205259.649246:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205259.649250:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4531 00000100:00080000:2.0:1788205259.649257:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205259.649261:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205259.649265:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205259.649279:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205259/real 1788205259] req@ffff90234598f800 x1875064443154432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205259.649280:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598e300 x1875064443154816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205259.649309:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f800 x1875064443154432/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205259.649321:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205259.649325:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205259.649329:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443154432:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788205259.649350:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443154560:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788205259.649368:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205259.649378:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205259/real 1788205259] req@ffff90234598ce00 x1875064443154560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205259.649390:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598ce00 x1875064443154560/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205259.649398:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205259.649401:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205259.649404:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443154560:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788205259.649409:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443154688:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205259.649423:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205259.649431:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205259/real 1788205259] req@ffff90234598f480 x1875064443154688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205259.649441:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f480 x1875064443154688/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205259.649449:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205259.649453:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205259.649456:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443154688:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788205259.649461:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443154816:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205259.649477:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205259.649486:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205259/real 1788205259] req@ffff90234598e300 x1875064443154816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205259.649496:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598e300 x1875064443154816/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205259.649504:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205259.649507:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205259.649510:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443154816:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205260.482258:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205261.505013:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205262.529029:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205263.553029:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205264.577099:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205264.769104:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205264.769114:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205264.769128:0:38:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205264.769142:0:38:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4536 00000100:00080000:2.0:1788205264.769153:0:38:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205264.769160:0:38:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:2.0:1788205264.769207:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c380 x1875064443154944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205264.769234:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205264.769238:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205264.769240:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205264.769244:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4536 00000100:00080000:2.0:1788205264.769247:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205264.769250:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205264.769259:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205264.769270:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598d180 x1875064443155072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205264.769289:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788205264.769295:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443154944:192.168.201.130@tcp:250:kworker.0 00000100:00080000:2.0:1788205264.769295:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205264.769297:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205264.769304:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4536 00000100:00080000:2.0:1788205264.769308:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205264.769311:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205264.769317:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205264.769334:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598c700 x1875064443155200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205264.769351:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205264.769354:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205264.769356:0:38:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205264.769360:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4536 00000100:00080000:2.0:1788205264.769365:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205264.769368:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:2.0:1788205264.769373:0:38:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788205264.769380:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:2.0:1788205264.769384:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff90234598f100 x1875064443155328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205264.769477:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205264/real 1788205264] req@ffff90234598c380 x1875064443154944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205264.769495:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c380 x1875064443154944/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205264.769505:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205264.769509:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205264.769512:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443154944:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788205264.769536:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443155072:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788205264.769573:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205264.769586:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205264/real 1788205264] req@ffff90234598d180 x1875064443155072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205264.769598:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598d180 x1875064443155072/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205264.769606:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205264.769609:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205264.769613:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443155072:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788205264.769621:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443155200:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205264.769638:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205264.769647:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205264/real 1788205264] req@ffff90234598c700 x1875064443155200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205264.769658:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598c700 x1875064443155200/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205264.769666:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205264.769669:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205264.769672:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443155200:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788205264.769677:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443155328:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205264.769691:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205264.769699:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205264/real 1788205264] req@ffff90234598f100 x1875064443155328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205264.769709:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff90234598f100 x1875064443155328/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205264.769717:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205264.769720:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205264.769723:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff90234598f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443155328:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788205265.601012:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205266.625051:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205267.651081:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205268.673083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205269.697084:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205269.889093:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205269.889103:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205269.889107:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205269.889115:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4541 00000100:00080000:0.0:1788205269.889126:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205269.889133:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788205269.889170:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443155456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205269.889202:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205269.889206:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205269.889208:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205269.889212:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4541 00000100:00080000:0.0:1788205269.889216:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205269.889220:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205269.889229:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205269.889253:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443155584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205269.889262:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205269.889265:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205269.889267:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205269.889271:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4541 00000100:00080000:0.0:1788205269.889274:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205269.889278:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205269.889282:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205269.889292:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443155712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205269.889312:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205269.889315:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205269.889317:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205269.889321:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4541 00000100:00080000:0.0:1788205269.889324:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205269.889327:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205269.889331:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205269.889341:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443155840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205269.889434:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443155456:192.168.201.130@tcp:250:kworker.0 00000800:00000100:1.0:1788205269.889490:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205269.890034:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205269/real 1788205269] req@ffff9023450b8700 x1875064443155456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205269.890053:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443155456/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205269.890063:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205269.890067:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205269.890070:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443155456:192.168.201.130@tcp:250:kworker.0 00000100:00100000:1.0:1788205269.890079:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443155584:192.168.201.130@tcp:38:kworker.0 00000800:00000100:1.0:1788205269.890103:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205269.890116:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205269/real 1788205269] req@ffff9023450bad80 x1875064443155584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205269.890128:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443155584/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205269.890135:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205269.890138:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205269.890141:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443155584:192.168.201.130@tcp:38:kworker.0 00000100:00100000:1.0:1788205269.890148:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443155712:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205269.890166:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205269.890176:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205269/real 1788205269] req@ffff9023450b9c00 x1875064443155712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205269.890341:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443155712/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205269.890357:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205269.890360:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205269.890363:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443155712:192.168.201.130@tcp:8:kworker.0 00000100:00100000:1.0:1788205269.890369:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443155840:192.168.201.130@tcp:8:kworker.0 00000800:00000100:1.0:1788205269.890491:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:1.0:1788205269.890503:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205269/real 1788205269] req@ffff9023450b9f80 x1875064443155840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205269.890621:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443155840/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205269.890630:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205269.890634:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:1.0:1788205269.890638:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443155840:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205270.723748:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205271.745059:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205272.770860:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205273.793061:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205274.817042:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205275.008970:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205275.008983:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205275.008987:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205275.008998:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4546 00000100:00080000:0.0:1788205275.009009:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205275.009016:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788205275.009052:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443155968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205275.009082:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205275.009086:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205275.009088:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205275.009092:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4546 00000100:00080000:0.0:1788205275.009095:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205275.009099:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205275.009109:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205275.009122:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443156096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205275.009128:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443155968:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788205275.009137:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205275.009142:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205275.009144:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205275.009149:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4546 00000100:00080000:0.0:1788205275.009153:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205275.009157:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205275.009164:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205275.009176:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443156224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205275.009193:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788205275.009194:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788205275.009199:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205275.009201:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205275.009206:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4546 00000100:00080000:0.0:1788205275.009210:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205275.009215:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205275.009220:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205275.009240:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443156352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205275.009291:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205275/real 1788205275] req@ffff9023450baa00 x1875064443155968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205275.009306:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443155968/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205275.009314:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205275.009317:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205275.009321:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443155968:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788205275.009343:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443156096:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788205275.009359:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205275.009367:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205275/real 1788205275] req@ffff9023450b9180 x1875064443156096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205275.009375:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443156096/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205275.009380:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205275.009382:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205275.009384:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443156096:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788205275.009387:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443156224:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788205275.009396:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205275.009401:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205275/real 1788205275] req@ffff9023450b8000 x1875064443156224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205275.009408:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443156224/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205275.009412:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205275.009414:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205275.009416:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443156224:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788205275.009421:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443156352:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788205275.009435:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205275.009445:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205275/real 1788205275] req@ffff9023450ba300 x1875064443156352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205275.009467:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443156352/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205275.009472:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205275.009474:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205275.009476:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443156352:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788205275.841010:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205276.865107:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205277.888988:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205278.914022:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205279.937049:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205280.128975:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205280.128984:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205280.128987:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205280.129006:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4551 00000100:00080000:0.0:1788205280.129016:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205280.129023:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788205280.129063:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450ba300 x1875064443156480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205280.129091:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205280.129095:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205280.129097:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205280.129101:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4551 00000100:00080000:0.0:1788205280.129106:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205280.129111:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205280.129119:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205280.129134:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8000 x1875064443156608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205280.129145:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788205280.129146:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443156480:192.168.201.130@tcp:250:kworker.0 00000100:00080000:0.0:1788205280.129150:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205280.129154:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205280.129159:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4551 00000100:00080000:0.0:1788205280.129163:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205280.129166:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205280.129172:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205280.129185:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9180 x1875064443156736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205280.129194:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205280.129198:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205280.129200:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205280.129203:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4551 00000100:00080000:0.0:1788205280.129207:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788205280.129212:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00080000:0.0:1788205280.129214:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205280.129219:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205280.129227:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450baa00 x1875064443156864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205280.129321:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205280/real 1788205280] req@ffff9023450ba300 x1875064443156480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205280.129339:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450ba300 x1875064443156480/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205280.129359:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205280.129363:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205280.129367:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450ba300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443156480:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788205280.129388:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443156608:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788205280.129408:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205280.129419:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205280/real 1788205280] req@ffff9023450b8000 x1875064443156608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205280.129429:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8000 x1875064443156608/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205280.129437:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205280.129440:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205280.129443:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443156608:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788205280.129448:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443156736:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788205280.129465:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205280.129475:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205280/real 1788205280] req@ffff9023450b9180 x1875064443156736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205280.129486:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9180 x1875064443156736/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205280.129494:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205280.129496:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205280.129499:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443156736:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788205280.129504:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443156864:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788205280.129519:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205280.129543:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205280/real 1788205280] req@ffff9023450baa00 x1875064443156864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205280.129555:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450baa00 x1875064443156864/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205280.129562:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205280.129565:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205280.129568:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450baa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443156864:192.168.201.130@tcp:8:kworker.0 00010000:00010000:1.0:1788205280.961056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205281.985729:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205283.009079:0:5599:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205284.033963:0:13084:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205285.057133:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205285.249028:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7ad49433-54fa-4e21-b756-859420d95603->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205285.249044:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205285.249049:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205285.249064:0:11804:0:(import.c:510:import_select_connection()) MGC192.168.201.130@tcp: connect to NID 192.168.201.130@tcp last attempt 4556 00000100:00080000:0.0:1788205285.249076:0:11804:0:(import.c:589:import_select_connection()) MGC192.168.201.130@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205285.249082:0:11804:0:(import.c:624:import_select_connection()) MGC192.168.201.130@tcp: import ffff902350c3d800 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000100:00100000:0.0:1788205285.249130:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9f80 x1875064443156992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205285.249168:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205285.249171:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205285.249172:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205285.249175:0:11804:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4556 00000100:00080000:0.0:1788205285.249178:0:11804:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205285.249181:0:11804:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff902350a11000: import ffff902348897000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205285.249195:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205285.249207:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b9c00 x1875064443157120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205285.249214:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205285.249216:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205285.249218:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205285.249222:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4556 00000100:00080000:0.0:1788205285.249224:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205285.249227:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff902350a11000: import ffff902348893000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205285.249230:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205285.249236:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450bad80 x1875064443157248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205285.249242:0:11804:0:(pinger.c:229:ptlrpc_pinger_process_import()) 35604eb4-baf9-45be-a20a-4883a2cd8cf9->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205285.249244:0:11804:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205285.249245:0:11804:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205285.249247:0:11804:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: connect to NID 192.168.201.130@tcp last attempt 4556 00000100:00080000:0.0:1788205285.249249:0:11804:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205285.249251:0:11804:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff902350a11000: import ffff902348894000 using connection 192.168.201.130@tcp/192.168.201.130@tcp 00000008:00100000:0.0:1788205285.249253:0:11804:0:(osc_request.c:3829:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34682934 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205285.249257:0:11804:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9023450b8700 x1875064443157376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205285.249361:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443156992:192.168.201.130@tcp:250:kworker.0 00000800:00000100:3.0:1788205285.249431:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205285.249485:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205285/real 1788205285] req@ffff9023450b9f80 x1875064443156992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205285.249498:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9f80 x1875064443156992/t0(0) o250->MGC192.168.201.130@tcp@192.168.201.130@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205285.249506:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902350c3d800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205285.249509:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205285.249512:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7ad49433-54fa-4e21-b756-859420d95603:2358:1875064443156992:192.168.201.130@tcp:250:kworker.0 00000100:00100000:3.0:1788205285.249516:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443157120:192.168.201.130@tcp:38:kworker.0 00000800:00000100:3.0:1788205285.249532:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205285.249540:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205285/real 1788205285] req@ffff9023450b9c00 x1875064443157120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205285.249547:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b9c00 x1875064443157120/t0(0) o38->lustre-MDT0000-mdc-ffff902350a11000@192.168.201.130@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205285.249551:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348897000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205285.249553:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205285.249554:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b9c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443157120:192.168.201.130@tcp:38:kworker.0 00000100:00100000:3.0:1788205285.249557:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443157248:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788205285.249567:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205285.249572:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205285/real 1788205285] req@ffff9023450bad80 x1875064443157248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205285.249578:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450bad80 x1875064443157248/t0(0) o8->lustre-OST0000-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205285.249584:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348893000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205285.249585:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205285.249587:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450bad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443157248:192.168.201.130@tcp:8:kworker.0 00000100:00100000:3.0:1788205285.249591:0:2358:0:(client.c:1906:ptlrpc_send_new_req()) Sending RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443157376:192.168.201.130@tcp:8:kworker.0 00000800:00000100:3.0:1788205285.249600:0:2358:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.130@tcp 00000100:00100000:3.0:1788205285.249604:0:2358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205285/real 1788205285] req@ffff9023450b8700 x1875064443157376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205285.249611:0:2358:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9023450b8700 x1875064443157376/t0(0) o8->lustre-OST0001-osc-ffff902350a11000@192.168.201.130@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205285.249616:0:2358:0:(import.c:65:import_set_state_nolock()) ffff902348894000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205285.249617:0:2358:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.130@tcp failed (-110) 00000100:00100000:3.0:1788205285.249618:0:2358:0:(client.c:2401:ptlrpc_check_set()) Completed RPC req@ffff9023450b8700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:35604eb4-baf9-45be-a20a-4883a2cd8cf9:2358:1875064443157376:192.168.201.130@tcp:8:kworker.0 00010000:00010000:2.0:1788205286.081164:0:13086:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 70764 lines, 70764 kept, 0 dropped, 0 bad.